MixFormerV2: Efficient Fully Transformer Tracking

In plain words

Visual object tracking means following one target box through a video, starting from where the target is in the first frame. Many recent trackers are accurate but slow. Part of the cost comes from heavy parts bolted onto a transformer: a dense head that scores many candidate positions, and a separate module that judges whether a frame is a good example of the target.

MixFormerV2 asks whether the transformer can do the whole job by itself. It adds a few special learnable tokens next to the image tokens. After the transformer runs, small shared heads read those tokens to produce the box, and another small head produces a confidence score.

To make the model smaller and faster, the authors train it to copy a larger, slower tracker (the teacher) and then remove some of its layers step by step instead of all at once. The full model is fast on a GPU, and a smaller variant is meant to run in real time on a CPU.

TL;DR

MixFormerV2 is a single-object tracker made only of transformer blocks, with no dense convolutional head and no separate score module. Four learnable prediction tokens are appended to the template and search tokens; a shared MLP turns them into distributions over each box edge, and a small MLP gives the confidence score. Distillation (dense-to-sparse, then progressive layer removal) makes it fast. Author-reported headline: 70.6% AUC on LaSOT at 165 FPS on GPU for the 8-layer B model. Before reimplementing, read Paper vs code: the released code differs from the paper in several loss and schedule details.

Problem

Transformer one-stream trackers are accurate, but inference efficiency, especially on CPU, limits deployment §1. The authors name three costs in their starting point, MixViT: a dense corner head that estimates box-corner probabilities on high-resolution maps §1, Table 1; a separate score module that uses RoI pooling, attention and an MLP to choose good online template samples §1; and the resulting architecture complexity §1.

They argue that a new lightweight backbone is unsuitable, because one-stream trackers depend heavily on large-scale pre-training. They therefore prune an existing backbone instead §3.2.2.

Assumptions: training uses the same data as MixFormer (TrackingNet, LaSOT, GOT-10k and COCO training splits) §4.1. Evaluation is on single-object tracking benchmarks (LaSOT, LaSOT-ext, TrackingNet, UAV123, TNL2K, VOT2020/2022, GOT-10k) §4.3, app. S.4.

Method

Overview

The model is a plain ViT-style backbone over one sequence: the template tokens, the search-region tokens and four learnable prediction tokens §3.1, fig. 2. It has no convolutional head. Two MLP heads sit on the prediction tokens: one for the box, one for the score §3.1. The paper does not name a number of attention heads; the code uses 12 heads for the ViT-B configuration mixformer2_vit.py.

Prediction-token mixed attention (P-MAM)

The attention is asymmetric §3.1, eq. 1:

• Template queries attend only to template keys and values.
• Search-region queries and prediction-token queries attend to the concatenation of template, search and prediction-token keys and values.

The paper gives the attention as a softmax over scaled dot products with 1/sqrt(d) eq. 1. The code applies the scale per head mixformer2_vit.py.

\mathrm{Atten}_t = \mathrm{Softmax}\!\left(\tfrac{q_t k_t^{\top}}{\sqrt{d}}\right) v_t

\mathrm{Atten}_{s,e} = \mathrm{Softmax}\!\left(\tfrac{q_{s,e} k_{tse}^{\top}}{\sqrt{d}}\right) v_{tse} eq. 1

Direct prediction from tokens

• The four prediction tokens each regress one edge of the box: left, top, right and bottom §3.1.
• One MLP is shared across the four tokens §3.1, §3.2.1, eq. 2.
• Each edge is a probability distribution over discrete positions; the box edge is the expectation under that distribution §3.2.1, eq. 3.
• The score head averages the four tokens and passes the average through a two-layer MLP to get a real-valued confidence score §3.1. The paper's stated order is average first, then MLP. The code does the reverse; see Paper vs code.

\hat{P}_X(x) = \mathrm{MLP}(\mathrm{token}_X), \quad X \in \{\mathcal{T}, \mathcal{L}, \mathcal{B}, \mathcal{R}\} eq. 2

B_X = \mathbb{E}_{\hat{P}_X}[X] = \int x\, \hat{P}_X(x)\, dx eq. 3

The paper does not state how many positions each edge distribution has, nor how they are spaced. The code uses a fixed number of bins with equal pixel spacing head.py.

Dense-to-sparse distillation (stage 1)

• A dense-corner MixViT teacher gives two 2-D corner maps, one for top-left and one for bottom-right §3.2.1.
• Each teacher map is turned into four 1-D marginals, one per edge §3.2.1, eq. 4.
• The student's four edge distributions are matched to these marginals with KL divergence §3.2.1, eq. 5.

L_{\text{log}} = \sum_{X} L_{\text{KL}}(\hat{P}_X, P_X) eq. 5

Deep-to-shallow distillation (stage 2) and progressive depth pruning

• The student starts as a full copy of the teacher, so the initial representations are as similar as possible §3.2.2.
• A feature-mimicking loss matches selected student and teacher layer outputs with an L2 term §3.2.2, eq. 6.
• For each layer to be eliminated, its residual update is scaled by a factor gamma §3.2.2, eq. 8.
• Gamma follows a cosine curve over the first m = 40 epochs and is 0 afterwards §3.2.2, eq. 9. When gamma reaches 0 the layer is an identity map and is removed.

L_{\text{feat}} = \sum_{(i,j)\in\mathcal{M}} L_2(F_i^S, F_j^T) eq. 6

\gamma(t) = \begin{cases} 0.5\,(1 + \cos(\tfrac{t}{m}\pi)), & t \le m \\ 0, & t > m \end{cases} eq. 9

The paper does not name the layers it removes for the 12-to-8 step; it says only "certain layers" §3.2.2. Other stated details:

• An intermediate teacher is used when the student is very shallow, for example 12 → 8 → 4 layers §3.2.2, §4.1. The overall pipeline is in fig. 3; the elimination process is in fig. 4.
• For the CPU variant, the hidden width of the MLP is pruned by slicing the teacher's weights and then distilling. The paper writes the slicing as w' = w[:d1', :d2'] §3.2.2, MLP reduction.

Training objective

L = \lambda_1 L_1 + \lambda_2 L_{\text{ciou}} + \lambda_3 L_{\text{log}} + \lambda_4 L_{\text{feat}} eq. 10

The paper sets lambda1 = 5 (L1), lambda2 = 2 (CIoU), lambda3 = 1 (logits) and lambda4 = 0.2 (features) §4.1, §3.3. The score head is trained afterwards for 50 epochs §3.3.

Pseudocode: forward pass §3.1 eq. 1 eq. 2

Written from the paper's description. The code's score head differs; see Paper vs code.

# MixFormerV2 forward pass  [§3.1] [eq. 1] [eq. 2]
# Input:  template tokens (B, Nt, d), search tokens (B, Ns, d)
# Output: box (B, 4) in [0, 1], score (B,) logit

def forward(template, search):
    # Append the four learnable prediction tokens  [§3.1]
    predict_tokens = learned_tokens(4)                 # (B, 4, d)
    tokens = concat(template, search, predict_tokens)  # (B, Nt+Ns+4, d)

    # Asymmetric mixed attention: template queries stay inside the
    # template; search and prediction queries see all keys  [§3.1] [eq. 1]
    for layer in backbone_layers:
        tokens = layer(tokens, template_len=Nt)

    tokens_pred = tokens[:, -4:]                       # (B, 4, d)

    # One shared MLP per box edge; distribution over bins  [§3.1] [eq. 2]
    # Bin count and spacing are not stated in the paper.
    for edge in (left, top, right, bottom):
        dist_edge = softmax(mlp(tokens_pred[:, edge]))   # (B, bins)
        # Expected value of the distribution is the box edge  [eq. 3]
        coord_edge = sum(dist_edge * bin_positions)      # (B,)

    box = stack(coord_left, coord_top, coord_right, coord_bottom)

    # Paper order: average the four tokens, then a two-layer MLP  [§3.1]
    score = score_mlp(mean(tokens_pred, dim=1))          # (B, 1)
    return box, score

Pseudocode: stage 1 loss §3.2.1 eq. 4 eq. 5 eq. 10

# Stage 1: dense-to-sparse distillation  [§3.2.1] [eq. 4] [eq. 5]
# Lambda values from [§4.1]; the code uses other weights, see Paper vs code.

def stage1_loss(student, teacher, gt_box):
    # Teacher corner maps become four 1-D edge marginals  [eq. 4]
    teacher_edges = marginals(teacher.corner_maps)   # 4 x (B, bins)

    # KL between each student edge distribution and teacher marginal  [eq. 5]
    loss_logits = 0
    for edge in (left, top, right, bottom):
        loss_logits = loss_logits + kl(student.dist[edge], teacher_edges[edge])

    # L1 and CIoU on the predicted box  [§3.3] [eq. 10]
    loss_box = 5.0 * l1(student.box, gt_box) + 2.0 * ciou(student.box, gt_box)

    # lambda3 = 1 in the paper  [§3.3]
    return loss_box + 1.0 * loss_logits

Pseudocode: progressive layer removal §3.2.2 eq. 8 eq. 9

# Progressive depth pruning  [§3.2.2] [eq. 8] [eq. 9]
# The student starts as a copy of the teacher, then layers are removed.

def gamma_at(epoch, m=40):
    # Cosine schedule: 1 at the start, 0 after m epochs  [eq. 9]
    if epoch > m:
        return 0.0
    return 0.5 * (1.0 + cos(pi * epoch / m))

def eliminated_block(x_prev, gamma):
    # Attention and feed-forward outputs as in the paper  [eq. 8]
    attn_out = attention(x_prev)
    ffn_out = ffn(attn_out + x_prev)
    # Scale the update; at gamma = 0 the block is an identity map  [eq. 8]
    return x_prev + gamma * (ffn_out + attn_out)

Pseudocode: dynamic template update at test time §4.1

The paper states only the update interval and the choice of the highest-scoring sample. Its threshold and reset rules are not stated.

# Online template update  [§4.1]
# Interval 200 frames; keep the highest-scoring crop as the candidate.

def update_state(frame_index, crop, score, state):
    # Track the highest-scoring crop seen so far  [§4.1]
    if score > state.best_score:
        state.best_score = score
        state.best_crop = crop

    # Every 200 frames, the online template is refreshed  [§4.1]
    if frame_index % 200 == 0:
        state.online_template = state.best_crop
    return state

Hyperparameters

Values given in the paper are cited to the paper. Values taken only from the released code are marked "code only".

Results

Author-reported. Setup: single-object tracking benchmarks; models and speeds are as stated in §4.1. The speed column comes from the paper; the paper does not state the hardware or source for the baselines' speeds.

Main comparison (B variant) tab. 5

The table also reports normalised precision for LaSOT and TrackingNet and precision for TNL2K and UAV123. These are omitted here for length; see tab. 5.

The text reports OSTrack gaps of 1.5% AUC on LaSOT and 2.4% AUC on TNL2K §4.3. The TNL2K gap is inconsistent with tab. 5; see Pitfalls.

CPU-real-time comparison (S variant) tab. 6

The paper's text reports S beating FEAR-L by 2.7% AUC on LaSOT and STARK-Lightning by 2.0% §4.3. The STARK-Lightning row is omitted here because its values do not fill the table's columns; see Pitfalls.

VOT and GOT-10k tab. 4 tab. 7

• Trained on LaSOT, TrackingNet, GOT-10k and COCO tab. 7 caption. VOT2022 comparisons are in tab. 4 and VOT2020/GOT-10k in tab. 7.

Ablations (LaSOT AUC, author-reported) tab. 3 tab. 8

In the S route table, the first row is the teacher, and the last row is the S model's settings. The paper labels default choices with gray cells, which the source text does not reproduce.

Computation of the localisation heads (appendix): the paper's count for the four-MLP token head is 2,580,480 (about 2.5M) operations; for the pyramidal corner head it is 3,902,587,776 (about 3.9B) app. S.5. The paper omits bias terms and normalisation from both counts app. S.5. The paper's FLOP-based numbers in tab. 1 are 22.5 GFLOPs for the token head and 26.2 to 27.2 for the corner heads at 8 layers.

Paper vs code

Code: official repository linked from the paper, github.com/MCG-NJU/MixFormerV2. Cloned at commit 10414383d20de5587b74ffcc07f4b5cc2bec2449. Licence: MIT (repository LICENSE file). Code paths below are relative to the repository root.

Discrepancies, each cited to both sides:

1. Score head order. The paper averages the four prediction tokens and then applies the MLP §3.1. The code applies the MLP to each token and averages the outputs head.py. The online model calls it as score_head(reg_tokens) mixformer2_vit_online.py.

2. Logits loss scale. The paper sums the KL terms over the four edges eq. 5. Both distillation actors divide the sum by 4 mixformer_distill_st1.py mixformer_distill_st2.py.

3. Logits loss weight. The paper sets lambda3 = 1 §3.3, §4.1. The distillation configs set CORNER_WEIGHT: 5.0 student_288_depth12.yaml student_288_depth12to8.yaml. The default config value is 1.0 config.py. The configs are what the training scripts read.

4. Feature loss. The paper uses a sum of L2 terms over matched layer pairs eq. 6. The code uses F.mse_loss, which averages over the stacked pairs mixformer_distill_st2.py. The matched layers come from DISTILL_LAYERS_* in the configs, not from the paper student_288_depth12to8.yaml.

5. Learning rate. The paper gives 1e-4, reduced to 1e-5 after 400 epochs §4.1. The distillation configs use LR: 0.0004 student_288_depth12.yaml. The scheduler is StepLR(optimizer, LR_DROP_EPOCH) with its default decay factor; the DECAY_RATE key is not read base_functions.py. The backbone trains at 0.1 times the head's rate base_functions.py. The paper does not mention this.

6. Epochs of the 12-to-8 step. The paper says 500 epochs for each stage §4.1, app. S.2. The 12-to-8 config sets EPOCH: 45 student_288_depth12to8.yaml. The trainer hard-codes 40 elimination epochs ltr_trainer.py.

7. Gamma schedule offset. The paper's gamma runs over epochs 0 to m eq. 9. The code uses min(epoch + 1, 40), so gamma is 0 from epoch index 39 onward, one epoch earlier than a literal reading of eq. 9 ltr_trainer.py.

8. Residual scaling. The paper scales the combined FFN and attention update eq. 8. The code scales each residual branch separately: attention first, then FFN on the updated stream mixformer2_vit_stu.py. Both give the identity at gamma = 0.

9. Layers removed. The paper does not name them for the B route §3.2.2. The 12-to-8 config removes zero-based blocks 0, 3, 6 and 9, i.e. the 1st, 4th, 7th and 10th layers student_288_depth12to8.yaml. Their parameters are frozen base_functions.py.

10. Score-head training length. The paper trains the score head for 50 epochs §3.3. The online config sets EPOCH: 40 288_depth8_score.yaml.

11. Teacher input. The paper says the S teacher takes a 224 search input §4.1. The code's resize calls for the teacher input are commented out, so the teacher receives the student's crops mixformer_distill_st1.py.

12. Online update rules. The paper states the update interval and the highest-scoring selection §4.1. The code adds a sigmoid score threshold of 0.5 and a decay of the running maximum, which defaults to 1.0 (so no decay) mixformer2_vit_online.py. Neither rule is in the paper.

13. Head details not in the paper. The box head is a two-layer MLP with LayerNorm and ReLU, applied to each token head.py. The prediction tokens get a learnable, zero-initialised positional embedding mixformer2_vit_online.py. The number of distribution bins is set per config (72 in the B distillation config student_288_depth12.yaml, 96 in the online configs 288_depth8_score.yaml).

14. MLP reduction. The paper describes the weight slicing w' = w[:d1', :d2'] §3.2.2. I searched the training and model code for slicing patterns and did not find it. The S config builds a 4-layer model with MLP ratio 1 224_depth4_mlp1_score.yaml, but I could not see how its weights are initialised. Treat this as not verified.

Official code exists; the discrepancies above are the ones I found by reading the training and model code and the configs. I did not run the code.

Pitfalls

• Inconsistent headline number. The paper's abstract and §4.3 text give 56.7% AUC on TNL2K for B. Table 5 gives 57.4%. The arXiv listing abstract gives 57.4%. The §4.3 text ("2.4% AUC over OSTrack") matches 56.7%, not the table. Use the table values, and check the source before quoting.
• Conflicting convolution count. The introduction says the dense corner head has ten convolutional layers §1. The appendix counts 24 convolutions for the pyramidal corner head app. S.5.
• Undefined details. The paper does not give the number of distribution bins, the crop factors, the number of attention heads, the optimizer betas, warm-up, or the data sampling interval. The code's values are listed in Hyperparameters and marked "code only". Do not treat them as paper facts.
• Score head. The paper's average-then-MLP and the code's MLP-then-average give different outputs. Implement one deliberately and say which.
• Score threshold and decay. The 0.5 threshold and the decay setting change which templates get stored. Neither is in the paper.
• Stage cost. Three distillation stages take about 43, 42 and 35 hours on 8 RTX 8000 GPUs app. S.3. The paper calls this a limitation app. S.2. In the code, the removed blocks' parameters are frozen during the removal stage base_functions.py.
• Initialisation of the student. The paper does not state how the 12-layer student is initialised for stage 1. The code uses an MAE checkpoint path for the depth-12 config student_288_depth12.yaml, and the 12-to-8 config uses a placeholder path student_288_depth12to8.yaml.
• Speed numbers. GPU FPS is measured on RTX 8000. CPU FPS is measured on a Xeon Gold 6230R §4.1. The paper does not state the hardware or source of the baselines' speed numbers. Compare speeds only within the same table.
• Table row missing. In tab. 6, the STARK-Lightning row has one fewer value than the header; it is omitted from this card.
• Broader impact. The authors name illegal positioning and surveillance of people as a concern, and failure in complex real-world scenes as a risk app. S.1.

Lineage

• Builds on: MixFormer (MixViT), an earlier one-stream tracker with dense corner heads and a score prediction module §1, §3.1. The paper's own reference list names it as "MixFormer: End-to-end tracking with iterative mixed attention" [bibliography].
• Compared against: TransT, STARK, MixFormer, OSTrack, SimTrack, SwinTrack, SBT, ToMP, DiMP, HCAT, E.T.Track, FEAR, LightTrack and CSWinTT §4.3, tab. 5, tab. 6.
• Superseded by: not stated in the content of this wiki.