Visual object tracking

Overview

Visual object tracking follows objects through the frames of a video. Two tasks are covered: single-object tracking (SOT), where one target is followed, and multi-object tracking (MOT), where several objects are followed with a consistent identity in each frame 2105.01922 §1 2105.01922 §4.2 2105.01922 §4.3. A multi-object tracker also has to decide when a target starts and when it ends, and which detection belongs to which target in each frame 1604.03635 § Problem.

Eight paper cards list this slug, so this page covers eight papers' task definitions and results, not the field. They are a benchmark for people and boats in open water, filmed from drones 2105.01922 §3; an online tracker evaluated on pedestrians in the MOTChallenge 2015 benchmark 1604.03635 § Benchmark results; a sports dataset with a tracker that adds appearance matching 2304.05170 §3 2304.05170 §4.2; an end-to-end transformer that tracks several objects and predicts masks 2101.02702 §3.1 2101.02702 §4.2; an RGB-thermal single-object tracker built from adapters on a frozen network 2312.10611 §3.2; a social-distancing monitor that uses a people detector and DeepSORT 2005.01385 §IV-A; StrongSORT, an upgraded DeepSORT with two optional post-processing steps 2202.13514 §III-B 2202.13514 §IV; and SimTrack, a single-object tracker that runs one ViT over the exemplar and search frames together 2203.05328 §3.2. The variants below are the ones these papers evaluate. Each paper's numbers come from its own benchmarks and protocol. Its other trackers are baselines or compared methods in that paper's table 2105.01922 §5.2 2105.01922 §5.3 1604.03635 § Benchmark results 2312.10611 tab. 1.

Detection is a separate step that feeds many trackers. The SeaDronesSee paper uses a detector inside its MOT baselines and studies it separately 2105.01922 §5.1 2105.01922 §5.3. TrackFormer is the exception in this set: one transformer detects new objects and carries existing tracks in the same attention pass, so there is no separate association step 2101.02702 §1 2101.02702 §3.2. The RNN tracker takes the benchmark's provided detections as input, and the authors say that a better detector improves tracking results in general [arxiv:1604.03635 § Benchmark results]. The SportsMOT tracker uses a YOLOX detector trained on CrowdHuman and then on SportsMOT 2304.05170 §5.1. StrongSORT is also tracking by detection, with a YOLOX-X detector whose training schedule follows ByteTrack 2202.13514 §III-B 2202.13514 §V-A. Detection is covered on object detection.

Variants

Single-object tracking (DiMP, PrDiMP and ATOM families)

The single-object trackers are the DiMP and PrDiMP families and ATOM. The paper reports success and precision curves on the SOT clips. In those curves, the DiMP and PrDiMP families outperform ATOM, and PrDiMP50 slightly trails DiMP50 2105.01922 §5.2 2105.01922 fig. 5. The paper gives no numeric table for SOT, only the curves and the text above 2105.01922 §5.2. DiMP18 and ATOM run at about 27.1 fps on one RTX 2080 Ti 2105.01922 §5.2. Card: SeaDronesSee.

Single-object tracking with two cameras (BAT adapters)

BAT tracks one target in a colour (RGB) stream and a thermal (TIR) stream. Both streams share a pre-trained ViT, and a bi-directional adapter in every layer adds a feature prompt from each stream into the other, in the attention and in the MLP stage 2312.10611 §3.2. No stream is fixed as dominant. The encoder and the head are frozen, and only the adapters are trained; the paper states 0.32M trainable parameters 2312.10611 §1 2312.10611 §3.2.

On the two RGB-T benchmarks, the paper reports the highest scores of the 19 methods in its table. On RGBT234 it gets 86.8 MPR and 64.1 MSR, and on LasHeR 70.2 PR and 56.3 SR. The closest runner-up is ViPT, with 83.5 and 61.7 on RGBT234 and 65.1 and 52.5 on LasHeR 2312.10611 tab. 1. In the adapter ablation on LasHeR, the BAT-RGB and BAT-TIR variants reach 55.4 and 54.8 SR, against 56.3 for the full model 2312.10611 tab. 2. Card: Bi-directional Adapter for Multi-modal Tracking.

Single-object tracking, joint exemplar and search ViT (SimTrack)

SimTrack puts the first-frame exemplar and the search frame into one ViT backbone, so the two images attend to each other in every layer. It has no Siamese correlation and no transformer interaction head; the search-token outputs of the last layer go straight to a corner predictor [arxiv:2203.05328 §3.2]. Exemplar tokens get their own learned position embedding, so the backbone can tell the two images apart 2203.05328 §3.2. A foveal window adds 64×64 central crops of the exemplar as extra input tokens for finer detail 2203.05328 §3.3. The paper does not describe any output for those tokens 2203.05328 §3.3.

Sim-B/16 reaches 69.3 AUC on LaSOT and 54.8 AUC on TNL2K, and Sim-L/14 reaches 70.5 and 55.6 2203.05328 tab. 1. Its component ablation on TNL2K adds the joint framework, the distinguishable position embedding and the foveal window in turn, from 52.2 to 54.8 AUC 2203.05328 tab. 8. The paper's headline speed, over 40 fps for Sim-B/16, is author-reported, and the hardware is not stated 2203.05328 §4.1. The main results use CLIP pre-training 2203.05328 §4.1; the pre-training comparison is in 2203.05328 tab. 5.

On TrackingNet, the paper gives 82.3 AUC for Sim-B/16 and 83.4 for Sim-L/14 2203.05328 tab. 1. On GOT-10k test, the table's AO is 68.6 for Sim-B/16 and 69.8 for Sim-L/14 2203.05328 tab. 3. The released code covers ViT-B/16 only and differs from the paper in several settings; see the card's Paper vs code section. Card: Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking.

Multi-object tracking, tracking-by-detection (Tracktor++)

Tracktor++ runs on top of a Faster R-CNN detector with a ResNet50 backbone, as the paper describes it 2105.01922 §5.3. On the swimmer task it reaches MOTA 55.0 and IDF1 69.6, and on the all-objects-in-water task MOTA 71.9 and IDF1 80.5 2105.01922 tab. 7 2105.01922 tab. 8. Card: SeaDronesSee.

Multi-object tracking, CenterNet-based (FairMOT)

FairMOT is evaluated in two backbone sizes, D34 and R34. The paper says its detector is a CenterNet with DLA34 or ResNet34 2105.01922 §5.3. FairMOT-D34 is the stronger of the two: MOTA 39.0 on the swimmer task and 36.5 on the all-objects task 2105.01922 tab. 7 2105.01922 tab. 8. Card: SeaDronesSee.

Multi-object tracking, appearance fusion on tracking-by-detection (MixSort)

MixSort is a plug-in for tracking-by-detection trackers. For each track, a Kalman prediction gives a search region. A MixFormer-style network compares the track's stored template with that region and gives a heatmap, and the heatmap value at each detection's centre becomes the visual similarity. That similarity is blended with the IoU score, and the blend is solved with the Hungarian algorithm 2304.05170 §4.1 2304.05170 §4.2 [arxiv:2304.05170 eq. 1]. The paper names ByteTrack and OC-SORT as the trackers it extends 2304.05170 §4.2.

On the SportsMOT test set, trained on train and validation, MixSort-OC reaches 74.1 HOTA against 73.7 for OC-SORT, and MixSort-Byte reaches 65.7 HOTA against 64.1 for ByteTrack 2304.05170 tab. 3. The gains are 0.4 to 1.6 HOTA on that table. On MOT17 with a private detector, the paper's text says MixSort is ahead of the compared trackers in HOTA, IDF1 and AssA 2304.05170 §5.2 2304.05170 tab. 5. In the ablation, the Kalman motion model lowers HOTA from 73.8 to 65.7 when the mixed similarity is used 2304.05170 tab. 6. Card: SportsMOT.

Multi-object tracking, DeepSORT in a people monitor (social distancing)

This pipeline counts groups of people in surveillance video. A YOLO v3 detector finds people, DeepSORT keeps their IDs, and people closer than a threshold of 90 to 170 pixels, in a three-value feature space, count as neighbours. The paper does not state how neighbours are joined into groups [arxiv:2005.01385 §IV-A]. DeepSORT combines a Kalman motion model, a Mahalanobis gate and a cosine appearance gate, and solves the matching with the Hungarian algorithm 2005.01385 §III-E. The paper compares three detectors on its own person dataset: YOLO v3 with mAP 0.846 at 23 FPS, Faster R-CNN with 0.969 at 3 FPS, and SSD with 0.691 at 10 FPS 2005.01385 tab. III. These are detector results. The paper reports no tracking metric and no accuracy figure for the grouping or the violation index 2005.01385 §VI. Card: Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques.

Multi-object tracking, online with recurrent networks (RNN_LSTM)

The tracker runs online, without looking at future frames [arxiv:1604.03635 § Our Approach]. A recurrent network predicts and updates each target's box, and an LSTM assigns detections to targets one target at a time. A per-target existence probability starts and ends tracks [arxiv:1604.03635 § Our Approach]. The motion model is shared across targets, and the paper trains it only on synthetic trajectories sampled from a simple model fitted to the MOTChallenge sequences 1604.03635 § Training Data. The state is the box (x, y, w, h), and no appearance model or optical flow is used [arxiv:1604.03635 § Preliminaries and Notation] 1604.03635 § Benchmark results.

On MOTChallenge 2015 test, the author-reported result is MOTA 19.0 and 165.2 FPS 1604.03635 tab. 2. The authors say it is not at the top of the online pedestrian ranking but is about two orders of magnitude faster 1604.03635 § Benchmark results. The speed is measured on "a standard CPU" with no model given 1604.03635 § Abstract. Card: Online Multi-Target Tracking Using Recurrent Neural Networks.

Multi-object tracking, DeepSORT upgraded with appearance and post-processing (StrongSORT)

StrongSORT keeps DeepSORT's tracking-by-detection structure and replaces its parts. It uses a stronger detector, BoT appearance features, a single exponentially smoothed appearance vector per track, camera-motion compensation by ECC, a Kalman update whose noise scales with detection confidence, and one global linear assignment that mixes appearance and motion costs instead of the matching cascade 2202.13514 §III-B 2202.13514 eq. 7 2202.13514 eq. 10. The paper claims no new algorithm in this section 2202.13514 §III-B.

Two optional steps make up StrongSORT++. AFLink links tracklets from frame numbers and positions only, with no appearance input, and GSI fills gaps by Gaussian-process smoothing of each track 2202.13514 §IV-A 2202.13514 §IV-B. GSI also rewrites observed boxes, not only the gaps 2202.13514 §IV-B. Both steps can be attached to other trackers.

On MOT17 validation, adding the components one at a time raises IDF1 and HOTA overall, with small dips in some rows (HOTA drops from 68.3 to 68.2 when EMA is added). The full set reaches 82.3 IDF1 and 69.6 HOTA, from the DeepSORT baseline's 77.3 and 66.3 2202.13514 tab. I. The ablation uses a validation split of the training sequences. On the same split, AFLink then GSI adds 1.0 IDF1 to the full StrongSORT row 2202.13514 tab. II. The card notes that the released AFLink training data may overlap with that validation split, so these ablation gains may be optimistic 2202.13514 §V-A dataset.py. On MOT17 test with private detections, the paper reports 63.5 HOTA and 78.5 IDF1 for online StrongSORT, and 64.4 HOTA, 79.5 IDF1 and 79.6 MOTA for StrongSORT++ 2202.13514 tab. IV. The online figure is the one to compare with online trackers. The StrongSORT++ figure includes the offline steps. Its speed is 7.5 FPS for the online row and 7.1 for StrongSORT++, on a single V100 that the paper uses for all runs 2202.13514 §V-A 2202.13514 tab. IV. Card: StrongSORT: Make DeepSORT Great Again.

Multi-object tracking, tracking-by-attention (TrackFormer)

TrackFormer is a DETR-style encoder-decoder with a ResNet-50 backbone 2101.02702 §4.2 2101.02702 app. A.1. It keeps two kinds of query. Object queries are a fixed learned set, here 500 for MOT17, and they start new tracks 2101.02702 §3.2 2101.02702 §4.2. Track queries are the previous frame's output embedding for each active track, so a track keeps its identity through the decoder without a motion or appearance model or graph optimization 2101.02702 §1 2101.02702 §3.2. A track starts when an object-query score exceeds 0.4, and ends when its score falls below 0.4 or when NMS at IoU 0.9 suppresses it. The paper's re-identification pool, which keeps an ended track for up to 5 frames, is described in the appendix 2101.02702 app. A.4.

Training uses frame pairs. The previous frame is sampled from a window around the current one, and the model sees simulated false negatives, which drop track queries, and false positives, which add background boxes as tracks 2101.02702 §3.3. Pretraining on CrowdHuman supplies simulated pairs built from single images 2101.02702 §4.2. The mask head used on MOTS20 is covered in the same section 2101.02702 §4.2.

On MOT17 test with private detections and CrowdHuman data only, the paper reports 74.1 MOTA and 68.0 IDF1, ahead of TraDeS and CenterTrack in its table 2101.02702 tab. 1. On MOT20 with the same data, it reports 68.6 MOTA and 65.7 IDF1 2101.02702 tab. 1. On MOTS20 test, the sMOTSA score is 54.9, against 40.6 for Track R-CNN 2101.02702 tab. 2. The authors report 7.4 FPS as a self-measured figure 2101.02702 tab. 1. Card: TrackFormer.

Task definitions that change the target set

The SeaDronesSee benchmark defines two MOT tasks on the same clips. MOT-Swimmer tracks floaters and swimmers only. MOT-All-Objects-In-Water also tracks boats, but not people on boats. All tracked objects share one class [arxiv:2105.01922 §4.3]. The task you choose sets which objects count as hits, misses and false positives in the scores 2105.01922 tab. 7 2105.01922 tab. 8.

SportsMOT tracks only the players on court. It has 240 overhead broadcast sequences from basketball, volleyball and football, split 45, 45 and 150 for train, validation and test 2304.05170 §3.1 2304.05170 §3.2. The authors note that its motion is faster and less regular than in pedestrian benchmarks, and that the main difficulty is association rather than detection 2304.05170 §1.

When to use which

The papers report on different data, so their numbers do not rank the trackers against each other. The SeaDronesSee benchmark gives accuracy numbers and a few speed numbers for SOT, but no speed for its MOT trackers, and no training settings for any tracker 2105.01922 §5.2 [arxiv:2105.01922 §5.3]. The RNN tracker gives speed on pedestrian data, measured on a CPU of unstated model 1604.03635 tab. 2. The BAT and SportsMOT numbers come from different benchmarks with different metrics. Read each table with those gaps in mind.

Implementation sketch

The SeaDronesSee card describes evaluation, not a tracker. The shape below is the flow the paper uses to score any submission: the test labels stay on the server, and the scores come from comparing predictions with them. The exact protocol for each metric is in the paper's cited sections, not restated here.

# Score tracker predictions for one clip set  [arxiv:2105.01922 §3.3] [§5]
# Input:  predicted boxes and identities per frame, per clip
# Output: task metrics

def score_submission(predictions, clips, task):
    # Test labels are withheld; predictions are uploaded for scoring  [§3.3]
    if clips.labels_are_withheld:
        return upload_to_server(predictions, task)
    # SOT: success and precision curves per clip  [§5.2]
    if task == "SOT":
        return success_and_precision(predictions, clips.labels)
    # MOT: MOT benchmark protocol on the chosen object set  [§4.3] [§5.3]
    return mot_protocol(predictions, clips.labels, task.object_set)

The RNN tracker follows a different shape, one step per frame. The steps are summarised here, with the paper's section for each [arxiv:1604.03635 § Our Approach]. The paper has no algorithm box, so this is a reconstruction.

# One online tracking step per frame  [arxiv:1604.03635 § Our Approach]
# Input:  target states, existence probabilities, detections, hidden state
# Output: updated states, existence (kept targets only), hidden state

def track_step(states, existence, detections, hidden):
    # Predict each target's next state from its state and the RNN memory
    predicted, hidden = motion_rnn(states, hidden)
    # Distance from each predicted state to each detection
    cost = pairwise_distance(predicted, detections)
    # LSTM assigns detections to targets, one target row at a time;
    # the extra column means "no detection"
    assignment = lstm_associate(cost)
    # Weighted update of each target from its assigned evidence
    states = weighted_update(predicted, detections, assignment)
    # Existence for the next frame; targets at or below 0.6 are dropped
    existence = existence_head(states, existence, detections, assignment)
    keep = existence > 0.6
    return states[keep], existence[keep], hidden

The TrackFormer paper's loop, one frame at a time, has the same shape as the RNN tracker's, but identity is carried by the query and not by an assignment step 2101.02702 §3.2 2101.02702 app. A.4. The sketch below omits the re-identification pool.

# Tracking by attention, one frame at a time  [arxiv:2101.02702 §3.2]
# Input:  video frames; score threshold 0.4; NMS IoU 0.9
# Output: per-frame boxes with track identities

def track_video(frames):
    tracks = []
    for frame in frames:
        # Decode previous track embeddings with the learned object queries
        outputs = detect(frame, track_queries=[t.embedding for t in tracks])
        # Keep a track only while its score stays above the threshold
        tracks = [t for t in tracks if outputs.score(t) > 0.4]
        # Start a track for each object query above the threshold
        tracks += new_tracks(outputs.object_queries, threshold=0.4)
        # Remove duplicate tracks by NMS at IoU 0.9
        tracks = nms_tracks(tracks, iou_threshold=0.9)
        emit(tracks)

Contradictions

Within the SeaDronesSee paper, the text and the table differ on one point: the text gives +6.4 AP50avg for the medium angle domain, and tab. 9 gives 39.3 − 32.7 = 6.6 2105.01922 §5.4 2105.01922 tab. 9. The paper also gives two altitude ranges for different rows of tab. 1: 5–260 m for detection, and 5–150 m for the single- and multi-object tracking rows [arxiv:2105.01922 tab. 1]. The paper does not say which clips reach which heights, so the tracking range cannot be confirmed from the card.

Within the RNN paper, the abstract gives a higher speed than the table: about 300 Hz in the abstract 1604.03635 § Abstract, and 165.2 FPS for the same method in tab. 2 1604.03635 tab. 2. Use the table value, and note that the hardware is not stated. Its test MOTA is 19.0 against 22.3 on the training set, and the paper does not discuss the gap 1604.03635 tab. 1 1604.03635 tab. 2.

Within the SportsMOT paper, the same MixSort-Byte run is reported with two slightly different scores: MOTA 96.2 and 2,472 IDs in tab. 3, against MOTA 96.1 and 2,469 IDs in tab. 6 and tab. 7. The abstract's dataset ratios also differ from the table: the abstract says about 3 times the boxes of MOT17, but tab. 1 gives about 5.6 times 2304.05170 tab. 1. The main-table fusion weight is not stated, and the released code's default of 0.6 is not one of the per-sport optima 2304.05170 tab. 4.

Within the BAT paper, the ablation puts BAT-Dual at 56.4 SR on LasHeR, above the full BAT model at 56.3 2312.10611 tab. 2. The number of LasHeR attributes is also given two ways: 19 in total with 7 new in §4.1.1, and 19 additional attributes in §4.3.3 2312.10611 §4.1.1 [arxiv:2312.10611 §4.3.3]. Check the dataset before citing either.

Within the social-distancing paper, the detector used with DeepSORT is named differently in two places: Faster R-CNN in §III-E and YOLO v3 in §IV. The final choice is YOLO v3 2005.01385 §V. The backbone for the detectors is named as Inception v2 in §V and as Darknet-53 for YOLO v3 in §III-D 2005.01385 §III-D 2005.01385 §V.

Within the TrackFormer paper, the MOTS20 mask-training table does not match the text. The text says that joint mask training improves IDF1 by 1.2. The two IDF1 values in the table differ by 1.2, but the table's marker sits on the row with the higher value (56.0), so the table does not show which row is the joint-training run 2101.02702 §4.4 2101.02702 tab. 4. Treat the row labels as unresolved. The paper's text also describes its false-positive and false-negative augmentations per query 2101.02702 §3.3, while the authors' released code draws batch-wide counts. The paper card's "Paper vs code" section lists these differences, so check which one you reproduce.

Within the StrongSORT paper and its released code, several settings differ. The paper sets tracker-level NMS to 0.8 2202.13514 §V-A, but the released tracker uses 1.0, which suppresses nothing; 0.8 is used at detection export 2202.13514 §V-A opts.py. The paper gives a matching distance of 0.45, while the code uses 0.4 for BoT and adds 0.05 when the motion cost is on 2202.13514 §V-A opts.py. The paper writes the NSA noise as (1 − c)R, but the code scales the standard deviation, which squares the factor in the covariance 2202.13514 eq. 9 [code: deep_sort/kalman_filter.py#L148-L151]. The paper's AFLink loss also differs from the code's two-class cross-entropy 2202.13514 eq. 11 [code: AFLink/train.py#L34]. Check the paper and the code separately before you reproduce either.

Within the SimTrack paper, Sim-B/16 is given 25.0 GFLOPs in tab. 1 and 23.4 GFLOPs in tab. 4 for the same joint model. The card does not resolve which is right 2203.05328 tab. 1 2203.05328 tab. 4. The text calls the GOT-10k score AUC, while the table labels it AO, and the text's 69.8 for Sim-L/14 is the table's AO value. Use AO 2203.05328 §4.2 2203.05328 tab. 3.

Across the papers, the numbers are not comparable. They use different datasets, different object classes, different metrics (for example MPR and MSR on RGBT234 against PR and SR on LasHeR) and different hardware for speed.

Papers

• SeaDronesSee: A Maritime Benchmark for Detecting Humans in Open Water: a UAV benchmark with SOT and MOT tasks, withheld test labels and author-reported tracker baselines.
• Online Multi-Target Tracking Using Recurrent Neural Networks: an online pedestrian tracker in which RNNs predict and update targets and an LSTM handles association, trained on synthetic trajectories.
• StrongSORT: Make DeepSORT Great Again: an upgraded DeepSORT with a stronger detector, appearance model and global matching, plus AFLink and GSI as optional post-processing.
• SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes: a sports tracking dataset with only on-court players tracked, and MixSort, an appearance-fusion plug-in for tracking-by-detection trackers.
• TrackFormer: Multi-Object Tracking with Transformers: an end-to-end transformer tracker in which track queries carry each object's identity from frame to frame, with a mask head for MOTS20.
• Bi-directional Adapter for Multi-modal Tracking: an RGB-thermal single-object tracker in which small bi-directional adapters let two streams prompt each other in a frozen ViT.
• Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking: a single-object tracker in which one ViT takes exemplar and search tokens together, with no Siamese correlation or transformer head.
• Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques: a people detector with DeepSORT and a distance-based grouping rule for surveillance video; the paper reports no tracking metric.