Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking

In plain words

Object tracking means following one target, such as a person or a car, through a video. The usual approach looks at the first frame, where the target is marked, and then searches each new frame for something that looks like it. Most recent trackers process the marked picture and the new picture in two separate branches, then add extra modules that compare them.

This paper's idea is to remove those extra comparison modules. The two pictures are cut into small squares, the squares from both pictures are placed in one row, and a single transformer reads the whole row at once. Because every square can look at every other square, the network learns to compare the two pictures as it goes, instead of through a hand-built step at the end.

The transformer can lose fine detail when it shrinks the picture, so the authors add a second, close-up view of the centre of the marked picture. The result is a simpler design with fewer custom parts, which the authors report as competitive with more specialised trackers.

TL;DR

• Replaces the Siamese backbone and the transformer interaction head of a STARK-style tracker with one ViT backbone that takes the exemplar and search tokens together §3.2.
• Attention between exemplar and search tokens runs in every layer, so no separate interaction module is needed §3.2, eq. 6.
• A foveal window adds 64×64 central crops of the exemplar as extra tokens for finer detail §3.3.
• Author-reported: ViT-B/16 gets 69.3% AUC on LaSOT and 54.8% on TNL2K; ViT-L/14 gets 70.5% and 55.6% tab. 1.
• The official code is MIT-licensed, but the shipped configs and tracker differ from the paper in several places. See "Paper vs code".

Problem

Siamese trackers encode the exemplar (the marked first frame) and the search frame with separate branches, and then compare them in an interaction head §1, fig. 1. Transformer heads raise accuracy, but they are customised and hand-designed, so they are hard to fit into a general architecture §1. Down-sampling in a vision transformer can split the target across patches and lose detail that localisation needs §1, §3.3. The paper assumes that a backbone pre-trained for classification can be used directly for tracking §3.2.

Method

Baseline (STARK-SV) §3.1. The authors first replace the ResNet-50 backbone of STARK-S with a ViT. Each image is cut into non-overlapping P×P patches, which are linearly projected to C-dimensional tokens and given positional embeddings §3.1. The exemplar and search are processed by separate backbone passes and then combined in a transformer head §3.1. A corner predictor turns the search features into a likelihood map for each of the box's two corners §3.1. Training minimises a weighted sum of generalised IoU loss and L1 loss on the predicted box §3.1, eq. 3.

Joint backbone (SimTrack) §3.2. The exemplar tokens e⁰ and search tokens s⁰ are concatenated along the sequence dimension and sent through the backbone together §3.2, eq. 4. In each layer the attention matrix has four blocks: exemplar-to-exemplar, exemplar-to-search, search-to-exemplar and search-to-search §3.2, eq. 5, eq. 6. Interaction therefore happens in every layer. The transformer head is removed. The search-token outputs of the last layer go straight to the corner predictor §3.2.

Distinguishable position embedding §3.2. The search tokens keep the pre-trained positional embedding, since the search input has the same size as in standard ViT training §3.2. Exemplar tokens get a separate learned embedding p_e, computed from each patch's grid position (i, j) and the fraction R_ij of the target area inside that patch, using two fully connected layers §3.2, eq. 7. The exemplar is smaller than the search image, and sharing one embedding would give the backbone no way to tell the two images apart §3.2.

Foveal window §3.3. A smaller crop is taken from the centre of the exemplar and cut into patches. Its grid lines fall on the centre of the exemplar's grid lines, so the foveal patches show different parts of the target than the normal patches do §3.3, fig. 3. The foveal patches use the same patch projection, and their position embeddings are computed as in eq. 7 §3.3. The foveal tokens are input-only: the paper's text does not describe any output for them §3.3. Sim-B/16 uses a 64×64 foveal crop §4.1, app. Training Details.

Training and inference §4.1, app. Training Details. Training pairs come from the same video, sampled within a frame range §3.1. The predictor is the same as in STARK-S §4.1. At inference there is no post-processing beyond a forward pass and coordinate mapping §4.1. Attention maps show the search features concentrating on the target across layers §4.3, fig. 4. More maps are in app. Visualization, fig. 5.

# SimTrack forward pass for one training pair  [§3.1] [§3.2] [§3.3]
# Input:  exemplar (B, 3, 112, 112), search (B, 3, 224, 224),
#         exemplar_box (B, 4) in exemplar coordinates
# Output: search features (B, N_x, C) for the corner predictor

def simtrack_forward(exemplar, search, exemplar_box):
    # Cut the exemplar into 16x16 patches and project each  [§3.1]
    exemplar_tokens = patch_embed(exemplar)           # (B, N_z, C), N_z = 49
    # Project the search image the same way, with the same projection  [§3.1]
    search_tokens = patch_embed(search)               # (B, N_x, C), N_x = 196
    # Add the learned exemplar position embedding  [§3.2] [eq. 7]
    exemplar_tokens = exemplar_tokens + exemplar_position(exemplar_box)
    # Crop 64x64 from the exemplar centre and project its patches  [§3.3]
    foveal_crop = center_crop(exemplar, size=64)
    foveal_tokens = patch_embed(foveal_crop)          # (B, 16, C)
    # Add position embeddings to the foveal tokens, as in eq. 7  [§3.3]
    foveal_tokens = foveal_tokens + foveal_position(exemplar_box)
    # Join the three token groups into one sequence  [§3.2] [§3.3]
    tokens = concat([exemplar_tokens, search_tokens, foveal_tokens], dim=1)
    # Run every layer over all tokens, so each one attends to all others  [§3.2]
    # [eq. 4] [eq. 5] [eq. 6]
    for layer in backbone_layers:
        tokens = layer(tokens)
    # Keep only the search outputs for the corner predictor  [§3.2]
    # The code also prepends a class token to the search group
    # [code: lib/models/stark/clipvit16.py#L153-L155]
    return tokens[:, N_z : N_z + N_x]

# Training loss for one predicted box  [§3.1] [eq. 3]
# Input:  pred_box, gt_box, both (B, 4) as (x1, y1, x2, y2) in [0, 1]
# Output: scalar loss

def training_loss(pred_box, gt_box):
    # GIoU term, weight 2  [eq. 3] [app. Training Details]
    giou_term = generalized_iou_loss(pred_box, gt_box)
    # L1 term on coordinates, weight 5  [eq. 3] [app. Training Details]
    l1_term = l1_loss(pred_box, gt_box)
    return 2.0 * giou_term + 5.0 * l1_term

# Per-video tracking loop  [§3.2] [§4.1]
# [code: lib/test/tracker/simtrack.py#L36-L75]
# Input:  frames (list of images), first_box (x, y, w, h) on frame 0
# Output: one box per frame

def track_video(frames, first_box):
    # Encode the exemplar crop once from the first frame  [§3.1]
    # [code: lib/test/tracker/simtrack.py#L38-L46]
    exemplar, exemplar_box = crop_exemplar(frames[0], first_box)
    boxes = [first_box]
    for frame in frames[1:]:
        # Crop the search region around the previous box  [§4.1]
        search = crop_search(frame, boxes[-1])
        # The joint backbone re-runs with the exemplar every frame  [§3.2]
        # [code: lib/test/tracker/simtrack.py#L65]
        pred = simtrack_forward_and_head(exemplar, search, exemplar_box)
        boxes.append(map_back_and_clip(pred, boxes[-1]))
    return boxes

Hyperparameters

Values are from the paper unless the "Where" column names code. "not stated" means the paper and the shipped code give no value.

Results

Author-reported, with the paper's setup. Test sets: LaSOT test has 280 sequences, TNL2K test 700, TrackingNet test 511 and UAV123 has 123 videos §4.2. TrackingNet and GOT-10k results come from evaluation servers §4.2. Top-3 colour marks from the paper are left out.

Table 1, state-of-the-art on LaSOT, TNL2K and TrackingNet test sets tab. 1. ◊ marks trackers with online updates. FLOPs cover backbone plus transformer head. For trackers without a head, FLOPs cover only the backbone.

Table 2, UAV123 tab. 2. Metrics are AUC and precision.

Table 3, GOT-10k_Test tab. 3. Metrics are AO and success rates at 0.5 and 0.75. The text calls the L/14 score 69.8 "AUC" §4.2, but the table row is AO.

Table 4, joint framework against the baseline STARK-SV with the same backbone, LaSOT and TNL2K tab. 4. Gains are in brackets.

Table 5, ViT-B/16 backbone with different pre-training weights, LaSOT and TNL2K tab. 5. MAE gives the best result.

The training-loss and AUC curves on LaSOT for STARK-SV and SimTrack §4.3 show that SimTrack reaches the same training loss in fewer epochs. The source places this plot in a table environment, so its number may differ in the PDF. The paper reports the same LaSOT AUC (66.8%) for SimTrack after 200 epochs as for STARK-SV after 500 §4.3.

Table 7, SimTrack with PVT-Medium and Swin-Base backbones tab. 7 against DiMP, TrDiMP, TransT and STARK-S. These backbone changes are not in the released code (see "Paper vs code").

Table 8, component analysis on TNL2K, AUC/Pre tab. 8. Base is STARK-SV with ViT-B/16.

Table 9, decoder layers added after the backbone, TNL2K AUC/Pre tab. 9. Adding decoder layers barely changes the score.

Table 10, share of backbone blocks with exemplar-search interaction, TNL2K AUC/Pre tab. 10. Fewer interacting blocks cost accuracy.

Table 11, input size for Sim-B/16 tab. 11, app. Input Resolution. Larger search inputs help.

Paper vs code

Official code: LPXTT/SimTrack at code_commit, MIT, linked from the abstract. Clone is raw/code/2203.05328. The configs and tracker are for ViT-B/16 only. Every difference below was checked by reading the code.

• Schedule. Paper: 500 epochs, batch 256, LR ×0.1 at epoch 400 app. Training Details. Shipped baseline.yaml: 50 epochs, batch 64, LR drop at 40 baseline.yaml baseline.yaml. Default config: 500, 32, 400 config.py. The shipped train.sh and the README training command use the GOT-10k-only config, not the four-dataset config train.sh README.md.
• Exemplar position input. Paper: (i, j) and R_ij, the fraction of target area in each patch §3.2, eq. 7. Code: the IoU between each patch box and the template target box, mapped by (IoU + 1)/2, plus the patch centre divided by 112 clipvit16.py clipvit16.py clipvit16.py. The paper does not say what R_ij means exactly, and the code does not use area.
• Position MLP. Paper: two fully connected layers, widths and activations not stated §3.2. Code: Linear 3→384, ReLU, Linear 384→768, Sigmoid, then rescaled ×2 − 1 clipvit16.py clipvit16.py.
• Foveal position embedding. Paper: computed as in eq. 7 for foveal patches §3.3. Code: the exemplar grid averaged with a copy shifted by one patch, then cropped to the foveal region, with no separate IoU for foveal patches clipvit16.py clipvit16.py.
• Foveal shift. Paper, B/32: the exemplar is shifted by 16 px, half the patch size, before the foveal crop app. Training Details. Code: the shifted branch runs only if (template − foveal) % (2 × foveal) = 0, with pixel sizes clipvit16.py. For the paper's B/32 sizes (128, 64) this is False, so the code takes the unshifted centre crop clipvit16.py. No B/32 config is shipped to confirm it.
• Head projection. Paper: the search features go straight to the STARK-S corner predictor §3.2, §4.1. Code: a Linear(768 → 256) bottleneck before the corner head simtrack.py simtrack.py.
• Search position embedding. Paper: pre-trained embedding used unchanged for search §3.2. Code: pre-trained grid bilinearly interpolated to the search grid, a no-op at 224/16 but active for other sizes clipvit16.py. Ablation sizes 320 and 384 app. Input Resolution would hit this path.
• Loss reduction. Paper: weights only eq. 3. Code: GIoU and L1 both averaged box_ops.py simtrack.py. If the GIoU call raises, the actor silently sets it to zero simtrack.py.
• Gradient clipping. Not in the paper. Code clips the norm at 0.1 baseline.yaml ltr_trainer.py.
• Transformer blocks. Paper: equations 1 to 2 are single-head and drop LayerNorm, FFN is unspecified §3.1. Code: the blocks are the CLIP visual transformer, imported from the external clip package and not in this repo clipvit16.py clipvit16.py.
• Ablation and extension code missing. Block-subset interaction (tab. 10), decoder ablation (tab. 9), and the Swin and PVT backbones (tab. 7) have no code in the repo. build_backbone_simtrack only builds CLIPVIT backbone.py. clipvit16.py has no block mask, and simtrack.py has no decoder. Sim-B/16* (four blocks removed) has no truncation code either.
• Inference. Paper: forward pass and coordinate mapping only §4.1. Code: the box is the mean over N predictions, which is 1 here, then clipped with a 10-px margin simtrack.py simtrack.py. The exemplar comes from the first frame and is not updated simtrack.py. The checkpoint path is a placeholder string, so the tracker does not load a model as shipped simtrack.py.
• Learning rate of the position MLP. Not in the paper. Code gives pos_fc the head rate, not the backbone rate base_functions.py.
• Sample interval and jitter. Not in the paper. Code: max interval 200, centre jitter 3.5, scale jitter 0.5 baseline.yaml baseline.yaml baseline.yaml.
• README versus paper. README: Python 3.8.15 README.md, paper: 3.6.9 §4.1. The README's arXiv link is 2107.02960, not this id README.md. README reports 70.6 on GOT-10k_Test for Sim-B/16 README.md, while tab. 3 gives AO 68.6 for the same backbone, and the README does not name a metric. Other README numbers (LaSOT 69.3, TNL2K 54.8, TrackingNet 82.3, UAV123 69.8) match the paper.

Pitfalls

• Joint attention is per frame. Exemplar and search share attention in every layer, so the exemplar features depend on each search frame and cannot be cached. The code re-runs the backbone with the exemplar on every frame simtrack.py. For ViT-B/16 at 112 and 224 the sequence has 262 tokens (derived: 49 exemplar, 16 foveal, 196 search, 1 class token). Speed claims (over 40 fps) are author-reported, and the hardware is not stated §4.1.
• Crop factor semantics. The paper writes the crop as "2² times the target box", which is an area. The code sets the side to factor × √(w·h), so factor 2.0 gives an area of 2² processing_utils.py. Reading 2² as the side length gives a crop twice as wide per side.
• Foveal outputs are discarded. Foveal tokens are added as input, and their outputs are sliced off before the head clipvit16.py. The paper does not describe their output.
• Position embedding inputs are ambiguous. The paper says R_ij is the fraction of target area in each patch. The code uses IoU with the template box, mapped to [0.5, 1] before the MLP. A literal implementation of the paper gives different embeddings.
• The exemplar stays fixed. At test time the exemplar and its box come from the first-frame init box simtrack.py. The code never updates them.
• Missing transformer details. The paper gives no FFN width, no activation, no head count and no LayerNorm placement. Eq. 1 drops LayerNorm and eq. 2 is single-head, but the code uses CLIP blocks with LayerNorm and multi-head attention. Taking the equations literally gives a different network.
• Numeric guards. The actor raises on NaN predictions simtrack.py and clamps the GT boxes to [0, 1] simtrack.py. The GIoU failure fallback to zero simtrack.py can hide training bugs, so log it if you reuse this code.
• Table and figure numbering is from source. Numbers were derived from the LaTeX source, not from a rendered PDF. Table 6 is the training-loss plot, which sits inside a table environment. Its number may differ in the compiled PDF.
• Duplicate label. fig:m0 is defined for Fig. 1 and again for the appendix figure §1, app. Visualization. Text references to it are ambiguous.
• Notation slip. The foveal patch count is written N_x* in §3.3, but the crop is from the exemplar and should be N_z*.
• Metric labels. GOT-10k is called "AUC" in the text and "AO" in tab. 3 §4.2. Use AO.
• Missing config for the headline L/14 result. Only ViT-B/16 configs are shipped. Sim-L/14 needs patch size 14 (the code's WINDOW_SZ) and exemplar 84 §4.1. Reproduce it from the paper's values, not from the repo's YAML.
• Supplementary details. "All other training details are the same as STARK-S and in the supplementary materials" §4.1. The supplement is not in the source, so STARK-S training details are missing from this card.

Lineage

• Builds on: STARK-S (arXiv:2103.17154, not in this wiki), whose ResNet-50 backbone and transformer head this paper replaces §3.1 §4.1. Also the ViT backbone §3.1 and the CLIP vision-branch weights §4.1.
• Compared against: the Siamese and transformer trackers in tab. 1 to 3, including TransT, TrDiMP, DiMP, KeepTrack and TransInMo tab. 1 tab. 2 tab. 3.
• Superseded by: not stated in this wiki.