--- kind: concept slug: visual-object-tracking title: Visual object tracking summary: >- Following one object (single-object tracking) or several with stable identities (multi-object tracking) through video. Covers eight paper cards: drone-benchmark SOT and MOT baselines, RGB-thermal adapters, a joint-ViT SOT tracker, sports, people-monitor, RNN, transformer and StrongSORT trackers. Trade-offs: accuracy against speed, motion against appearance, and the detector behind the tracker. tags: [computer-vision, object-tracking, benchmark, uav, data-association, rgb-thermal, sports, transformer, tracking-by-detection] papers: ["1604.03635", "2005.01385", "2101.02702", "2105.01922", "2202.13514", "2203.05328", "2304.05170", "2312.10611"] related: [object-detection, tracking-by-detection, gaussian-process, recurrent-neural-network, bipartite-matching, set-prediction, object-queries, focal-loss, vision-transformer, parameter-efficient-fine-tuning, siamese-network, positional-encoding] status: draft updated_at: 2026-11-01 --- ## Overview Visual object tracking follows objects through the frames of a video. Two tasks are covered: single-object tracking (SOT), where one target is followed, and multi-object tracking (MOT), where several objects are followed with a consistent identity in each frame [arxiv:2105.01922 §1] [arxiv:2105.01922 §4.2] [arxiv:2105.01922 §4.3]. A multi-object tracker also has to decide when a target starts and when it ends, and which detection belongs to which target in each frame [arxiv:1604.03635 § Problem]. Eight paper cards list this slug, so this page covers eight papers' task definitions and results, not the field. They are a benchmark for people and boats in open water, filmed from drones [arxiv:2105.01922 §3]; an online tracker evaluated on pedestrians in the MOTChallenge 2015 benchmark [arxiv:1604.03635 § Benchmark results]; a sports dataset with a tracker that adds appearance matching [arxiv:2304.05170 §3] [arxiv:2304.05170 §4.2]; an end-to-end transformer that tracks several objects and predicts masks [arxiv:2101.02702 §3.1] [arxiv:2101.02702 §4.2]; an RGB-thermal single-object tracker built from adapters on a frozen network [arxiv:2312.10611 §3.2]; a social-distancing monitor that uses a people detector and DeepSORT [arxiv:2005.01385 §IV-A]; StrongSORT, an upgraded DeepSORT with two optional post-processing steps [arxiv:2202.13514 §III-B] [arxiv:2202.13514 §IV]; and SimTrack, a single-object tracker that runs one ViT over the exemplar and search frames together [arxiv:2203.05328 §3.2]. The variants below are the ones these papers evaluate. Each paper's numbers come from its own benchmarks and protocol. Its other trackers are baselines or compared methods in that paper's table [arxiv:2105.01922 §5.2] [arxiv:2105.01922 §5.3] [arxiv:1604.03635 § Benchmark results] [arxiv:2312.10611 tab. 1]. Detection is a separate step that feeds many trackers. The SeaDronesSee paper uses a detector inside its MOT baselines and studies it separately [arxiv:2105.01922 §5.1] [arxiv:2105.01922 §5.3]. TrackFormer is the exception in this set: one transformer detects new objects and carries existing tracks in the same attention pass, so there is no separate association step [arxiv:2101.02702 §1] [arxiv:2101.02702 §3.2]. The RNN tracker takes the benchmark's provided detections as input, and the authors say that a better detector improves tracking results in general [arxiv:1604.03635 § Benchmark results]. The SportsMOT tracker uses a YOLOX detector trained on CrowdHuman and then on SportsMOT [arxiv:2304.05170 §5.1]. StrongSORT is also tracking by detection, with a YOLOX-X detector whose training schedule follows ByteTrack [arxiv:2202.13514 §III-B] [arxiv:2202.13514 §V-A]. Detection is covered on [object detection](/concepts/object-detection). ## Variants ### Single-object tracking (DiMP, PrDiMP and ATOM families) The single-object trackers are the DiMP and PrDiMP families and ATOM. The paper reports success and precision curves on the SOT clips. In those curves, the DiMP and PrDiMP families outperform ATOM, and PrDiMP50 slightly trails DiMP50 [arxiv:2105.01922 §5.2] [arxiv:2105.01922 fig. 5]. The paper gives no numeric table for SOT, only the curves and the text above [arxiv:2105.01922 §5.2]. DiMP18 and ATOM run at about 27.1 fps on one RTX 2080 Ti [arxiv:2105.01922 §5.2]. Card: [SeaDronesSee](/papers/2105.01922). ### Single-object tracking with two cameras (BAT adapters) BAT tracks one target in a colour (RGB) stream and a thermal (TIR) stream. Both streams share a pre-trained ViT, and a bi-directional adapter in every layer adds a feature prompt from each stream into the other, in the attention and in the MLP stage [arxiv:2312.10611 §3.2]. No stream is fixed as dominant. The encoder and the head are frozen, and only the adapters are trained; the paper states 0.32M trainable parameters [arxiv:2312.10611 §1] [arxiv:2312.10611 §3.2]. On the two RGB-T benchmarks, the paper reports the highest scores of the 19 methods in its table. On RGBT234 it gets 86.8 MPR and 64.1 MSR, and on LasHeR 70.2 PR and 56.3 SR. The closest runner-up is ViPT, with 83.5 and 61.7 on RGBT234 and 65.1 and 52.5 on LasHeR [arxiv:2312.10611 tab. 1]. In the adapter ablation on LasHeR, the BAT-RGB and BAT-TIR variants reach 55.4 and 54.8 SR, against 56.3 for the full model [arxiv:2312.10611 tab. 2]. Card: [Bi-directional Adapter for Multi-modal Tracking](/papers/2312.10611). ### Single-object tracking, joint exemplar and search ViT (SimTrack) SimTrack puts the first-frame exemplar and the search frame into one ViT backbone, so the two images attend to each other in every layer. It has no Siamese correlation and no transformer interaction head; the search-token outputs of the last layer go straight to a corner predictor [arxiv:2203.05328 §3.2]. Exemplar tokens get their own learned position embedding, so the backbone can tell the two images apart [arxiv:2203.05328 §3.2]. A foveal window adds 64×64 central crops of the exemplar as extra input tokens for finer detail [arxiv:2203.05328 §3.3]. The paper does not describe any output for those tokens [arxiv:2203.05328 §3.3]. Sim-B/16 reaches 69.3 AUC on LaSOT and 54.8 AUC on TNL2K, and Sim-L/14 reaches 70.5 and 55.6 [arxiv:2203.05328 tab. 1]. Its component ablation on TNL2K adds the joint framework, the distinguishable position embedding and the foveal window in turn, from 52.2 to 54.8 AUC [arxiv:2203.05328 tab. 8]. The paper's headline speed, over 40 fps for Sim-B/16, is author-reported, and the hardware is not stated [arxiv:2203.05328 §4.1]. The main results use CLIP pre-training [arxiv:2203.05328 §4.1]; the pre-training comparison is in [arxiv:2203.05328 tab. 5]. On TrackingNet, the paper gives 82.3 AUC for Sim-B/16 and 83.4 for Sim-L/14 [arxiv:2203.05328 tab. 1]. On GOT-10k test, the table's AO is 68.6 for Sim-B/16 and 69.8 for Sim-L/14 [arxiv:2203.05328 tab. 3]. The released code covers ViT-B/16 only and differs from the paper in several settings; see the card's Paper vs code section. Card: [Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking](/papers/2203.05328). ### Multi-object tracking, tracking-by-detection (Tracktor++) Tracktor++ runs on top of a Faster R-CNN detector with a ResNet50 backbone, as the paper describes it [arxiv:2105.01922 §5.3]. On the swimmer task it reaches MOTA 55.0 and IDF1 69.6, and on the all-objects-in-water task MOTA 71.9 and IDF1 80.5 [arxiv:2105.01922 tab. 7] [arxiv:2105.01922 tab. 8]. Card: [SeaDronesSee](/papers/2105.01922). ### Multi-object tracking, CenterNet-based (FairMOT) FairMOT is evaluated in two backbone sizes, D34 and R34. The paper says its detector is a CenterNet with DLA34 or ResNet34 [arxiv:2105.01922 §5.3]. FairMOT-D34 is the stronger of the two: MOTA 39.0 on the swimmer task and 36.5 on the all-objects task [arxiv:2105.01922 tab. 7] [arxiv:2105.01922 tab. 8]. Card: [SeaDronesSee](/papers/2105.01922). ### Multi-object tracking, appearance fusion on tracking-by-detection (MixSort) MixSort is a plug-in for tracking-by-detection trackers. For each track, a Kalman prediction gives a search region. A MixFormer-style network compares the track's stored template with that region and gives a heatmap, and the heatmap value at each detection's centre becomes the visual similarity. That similarity is blended with the IoU score, and the blend is solved with the Hungarian algorithm [arxiv:2304.05170 §4.1] [arxiv:2304.05170 §4.2] [arxiv:2304.05170 eq. 1]. The paper names ByteTrack and OC-SORT as the trackers it extends [arxiv:2304.05170 §4.2]. On the SportsMOT test set, trained on train and validation, MixSort-OC reaches 74.1 HOTA against 73.7 for OC-SORT, and MixSort-Byte reaches 65.7 HOTA against 64.1 for ByteTrack [arxiv:2304.05170 tab. 3]. The gains are 0.4 to 1.6 HOTA on that table. On MOT17 with a private detector, the paper's text says MixSort is ahead of the compared trackers in HOTA, IDF1 and AssA [arxiv:2304.05170 §5.2] [arxiv:2304.05170 tab. 5]. In the ablation, the Kalman motion model lowers HOTA from 73.8 to 65.7 when the mixed similarity is used [arxiv:2304.05170 tab. 6]. Card: [SportsMOT](/papers/2304.05170). ### Multi-object tracking, DeepSORT in a people monitor (social distancing) This pipeline counts groups of people in surveillance video. A YOLO v3 detector finds people, DeepSORT keeps their IDs, and people closer than a threshold of 90 to 170 pixels, in a three-value feature space, count as neighbours. The paper does not state how neighbours are joined into groups [arxiv:2005.01385 §IV-A]. DeepSORT combines a Kalman motion model, a Mahalanobis gate and a cosine appearance gate, and solves the matching with the Hungarian algorithm [arxiv:2005.01385 §III-E]. The paper compares three detectors on its own person dataset: YOLO v3 with mAP 0.846 at 23 FPS, Faster R-CNN with 0.969 at 3 FPS, and SSD with 0.691 at 10 FPS [arxiv:2005.01385 tab. III]. These are detector results. The paper reports no tracking metric and no accuracy figure for the grouping or the violation index [arxiv:2005.01385 §VI]. Card: [Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques](/papers/2005.01385). ### Multi-object tracking, online with recurrent networks (RNN_LSTM) The tracker runs online, without looking at future frames [arxiv:1604.03635 § Our Approach]. A recurrent network predicts and updates each target's box, and an LSTM assigns detections to targets one target at a time. A per-target existence probability starts and ends tracks [arxiv:1604.03635 § Our Approach]. The motion model is shared across targets, and the paper trains it only on synthetic trajectories sampled from a simple model fitted to the MOTChallenge sequences [arxiv:1604.03635 § Training Data]. The state is the box (x, y, w, h), and no appearance model or optical flow is used [arxiv:1604.03635 § Preliminaries and Notation] [arxiv:1604.03635 § Benchmark results]. On MOTChallenge 2015 test, the author-reported result is MOTA 19.0 and 165.2 FPS [arxiv:1604.03635 tab. 2]. The authors say it is not at the top of the online pedestrian ranking but is about two orders of magnitude faster [arxiv:1604.03635 § Benchmark results]. The speed is measured on "a standard CPU" with no model given [arxiv:1604.03635 § Abstract]. Card: [Online Multi-Target Tracking Using Recurrent Neural Networks](/papers/1604.03635). ### Multi-object tracking, DeepSORT upgraded with appearance and post-processing (StrongSORT) StrongSORT keeps DeepSORT's tracking-by-detection structure and replaces its parts. It uses a stronger detector, BoT appearance features, a single exponentially smoothed appearance vector per track, camera-motion compensation by ECC, a Kalman update whose noise scales with detection confidence, and one global linear assignment that mixes appearance and motion costs instead of the matching cascade [arxiv:2202.13514 §III-B] [arxiv:2202.13514 eq. 7] [arxiv:2202.13514 eq. 10]. The paper claims no new algorithm in this section [arxiv:2202.13514 §III-B]. Two optional steps make up StrongSORT++. AFLink links tracklets from frame numbers and positions only, with no appearance input, and GSI fills gaps by Gaussian-process smoothing of each track [arxiv:2202.13514 §IV-A] [arxiv:2202.13514 §IV-B]. GSI also rewrites observed boxes, not only the gaps [arxiv:2202.13514 §IV-B]. Both steps can be attached to other trackers. On MOT17 validation, adding the components one at a time raises IDF1 and HOTA overall, with small dips in some rows (HOTA drops from 68.3 to 68.2 when EMA is added). The full set reaches 82.3 IDF1 and 69.6 HOTA, from the DeepSORT baseline's 77.3 and 66.3 [arxiv:2202.13514 tab. I]. The ablation uses a validation split of the training sequences. On the same split, AFLink then GSI adds 1.0 IDF1 to the full StrongSORT row [arxiv:2202.13514 tab. II]. The card notes that the released AFLink training data may overlap with that validation split, so these ablation gains may be optimistic [arxiv:2202.13514 §V-A] [code: AFLink/dataset.py#L55-L64]. On MOT17 test with private detections, the paper reports 63.5 HOTA and 78.5 IDF1 for online StrongSORT, and 64.4 HOTA, 79.5 IDF1 and 79.6 MOTA for StrongSORT++ [arxiv:2202.13514 tab. IV]. The online figure is the one to compare with online trackers. The StrongSORT++ figure includes the offline steps. Its speed is 7.5 FPS for the online row and 7.1 for StrongSORT++, on a single V100 that the paper uses for all runs [arxiv:2202.13514 §V-A] [arxiv:2202.13514 tab. IV]. Card: [StrongSORT: Make DeepSORT Great Again](/papers/2202.13514). ### Multi-object tracking, tracking-by-attention (TrackFormer) TrackFormer is a DETR-style encoder-decoder with a ResNet-50 backbone [arxiv:2101.02702 §4.2] [arxiv:2101.02702 app. A.1]. It keeps two kinds of query. Object queries are a fixed learned set, here 500 for MOT17, and they start new tracks [arxiv:2101.02702 §3.2] [arxiv:2101.02702 §4.2]. Track queries are the previous frame's output embedding for each active track, so a track keeps its identity through the decoder without a motion or appearance model or graph optimization [arxiv:2101.02702 §1] [arxiv:2101.02702 §3.2]. A track starts when an object-query score exceeds 0.4, and ends when its score falls below 0.4 or when NMS at IoU 0.9 suppresses it. The paper's re-identification pool, which keeps an ended track for up to 5 frames, is described in the appendix [arxiv:2101.02702 app. A.4]. Training uses frame pairs. The previous frame is sampled from a window around the current one, and the model sees simulated false negatives, which drop track queries, and false positives, which add background boxes as tracks [arxiv:2101.02702 §3.3]. Pretraining on CrowdHuman supplies simulated pairs built from single images [arxiv:2101.02702 §4.2]. The mask head used on MOTS20 is covered in the same section [arxiv:2101.02702 §4.2]. On MOT17 test with private detections and CrowdHuman data only, the paper reports 74.1 MOTA and 68.0 IDF1, ahead of TraDeS and CenterTrack in its table [arxiv:2101.02702 tab. 1]. On MOT20 with the same data, it reports 68.6 MOTA and 65.7 IDF1 [arxiv:2101.02702 tab. 1]. On MOTS20 test, the sMOTSA score is 54.9, against 40.6 for Track R-CNN [arxiv:2101.02702 tab. 2]. The authors report 7.4 FPS as a self-measured figure [arxiv:2101.02702 tab. 1]. Card: [TrackFormer](/papers/2101.02702). ### Task definitions that change the target set The SeaDronesSee benchmark defines two MOT tasks on the same clips. MOT-Swimmer tracks floaters and swimmers only. MOT-All-Objects-In-Water also tracks boats, but not people on boats. All tracked objects share one class [arxiv:2105.01922 §4.3]. The task you choose sets which objects count as hits, misses and false positives in the scores [arxiv:2105.01922 tab. 7] [arxiv:2105.01922 tab. 8]. SportsMOT tracks only the players on court. It has 240 overhead broadcast sequences from basketball, volleyball and football, split 45, 45 and 150 for train, validation and test [arxiv:2304.05170 §3.1] [arxiv:2304.05170 §3.2]. The authors note that its motion is faster and less regular than in pedestrian benchmarks, and that the main difficulty is association rather than detection [arxiv:2304.05170 §1]. ## When to use which The papers report on different data, so their numbers do not rank the trackers against each other. The SeaDronesSee benchmark gives accuracy numbers and a few speed numbers for SOT, but no speed for its MOT trackers, and no training settings for any tracker [arxiv:2105.01922 §5.2] [arxiv:2105.01922 §5.3]. The RNN tracker gives speed on pedestrian data, measured on a CPU of unstated model [arxiv:1604.03635 tab. 2]. The BAT and SportsMOT numbers come from different benchmarks with different metrics. Read each table with those gaps in mind. | Situation | Use | Why | |---|---|---| | One target to follow through a video | DiMP or PrDiMP family, not ATOM | Higher success and precision curves in the paper; PrDiMP50 slightly trails DiMP50 [arxiv:2105.01922 §5.2] [arxiv:2105.01922 fig. 5] | | One target, RGB and thermal both available | BAT adapters | Highest of the 19 compared methods on RGBT234 and LasHeR [arxiv:2312.10611 tab. 1] | | One target in RGB video, a large pre-trained ViT is available, and per-frame compute is affordable | SimTrack with ViT-B/16 | 69.3 LaSOT AUC and 54.8 TNL2K AUC, against 66.8 and 52.2 for the STARK-SV baseline with the same backbone [arxiv:2203.05328 tab. 1] [arxiv:2203.05328 tab. 4]. The table lists 25.0G FLOPs [arxiv:2203.05328 tab. 1] | | Single target, and the exemplar features must be computed once and reused across frames | Not SimTrack as published | Exemplar and search share attention in every layer, so exemplar features change with each search frame [arxiv:2203.05328 §3.2] [arxiv:2203.05328 eq. 6] | | Single target, compute is tight, and fewer interacting blocks are acceptable | SimTrack with all blocks interacting | Interaction in half the blocks gives 52.3 TNL2K AUC and in a quarter 49.8, against 54.8 for all blocks [arxiv:2203.05328 tab. 10] | | Single target, ViT backbone with a choice of pre-training | MAE weights, if the self-supervised pre-training is available | 70.3 LaSOT AUC with MAE against 69.3 with CLIP, which the paper uses by default [arxiv:2203.05328 tab. 5] [arxiv:2203.05328 §4.1] | | Single target, and a higher search resolution is affordable | Search size 320 or 384 | LaSOT AUC 70.0 at 320 and 70.4 at 384, against 69.3 at 224 [arxiv:2203.05328 tab. 11] | | Reproducing the paper's training schedule | Paper values, not the released configs | The paper uses 500 epochs and batch size 256 [arxiv:2203.05328 app. Training Details]. The released configs use other values; see the card's Paper vs code section | | Frozen large tracker, little labelled RGB-T data | BAT-style adapters | Only the adapters are trained, 0.32M parameters [arxiv:2312.10611 §1]. Training took about 8 hours on 4 GPUs for LasHeR train [arxiv:2312.10611 §4.1.2] | | Either camera may be the more useful one, depending on conditions | BAT, not a fixed dominant modality | The paper's argument for two-way prompting [arxiv:2312.10611 §1]. Its single-modality adapter variants score 55.4 and 54.8 SR on LasHeR, against 56.3 [arxiv:2312.10611 tab. 2] | | Several people and boats, identities must hold | Tracktor++ | Highest MOTA and IDF1 on both MOT tasks [arxiv:2105.01922 tab. 7] [arxiv:2105.01922 tab. 8] | | False alarms are costly, misses are tolerable | FairMOT-D34 | 3,604 false positives against 7,271 for Tracktor++ on the swimmer task, but 9,445 false negatives against 3,550 [arxiv:2105.01922 tab. 7] | | Lost targets are costly | Tracktor++ | 3,550 false negatives and 85.5 recall on the swimmer task, against 9,445 false negatives for FairMOT-D34 [arxiv:2105.01922 tab. 7] | | Detector choice is open | Tracktor++ only if its detector can be reused | The paper says the detector may explain the advantage but did not test it [arxiv:2105.01922 §5.3] | | Players look alike, motion is fast and irregular | MixSort on ByteTrack or OC-SORT, with a fusion weight tuned for the sport | Small HOTA gains over the base trackers [arxiv:2304.05170 tab. 3]. Best fusion weight differs by sport: 0.8 basketball, 0.5 volleyball, 0.4 football [arxiv:2304.05170 tab. 4] | | Appearance fusion with a Kalman motion model | Check the motion model before adding it | Kalman lowers HOTA from 73.8 to 65.7 with mixed similarity on SportsMOT [arxiv:2304.05170 tab. 6] | | Online tracking with a tight CPU budget, and detections are given | RNN_LSTM tracker, if low accuracy is acceptable | 165.2 FPS author-reported with MOTA 19.0 on MOTChallenge 2015 test [arxiv:1604.03635 tab. 2] | | Several pedestrians, one trained network should handle detection and identity, and CrowdHuman-style data is available | TrackFormer in the private-detection setting | 74.1 MOTA and 68.0 IDF1 on MOT17 test with CrowdHuman data only [arxiv:2101.02702 tab. 1]. Training takes about two days on seven 32 GB GPUs [arxiv:2101.02702 §4.2] | | Detections must come from a fixed public detector | TrackFormer with the IoU filter, and compare only under that filter | Public-detection MOT17 test: 62.3 MOTA and 57.6 IDF1, while TMOH reports 62.8 IDF1 in the same table [arxiv:2101.02702 tab. 1]. The filter changes the numbers [arxiv:2101.02702 app. B.1] | | Masks are needed as well as boxes for people | TrackFormer's mask head | 54.9 sMOTSA on MOTS20 test, against 40.6 for Track R-CNN [arxiv:2101.02702 tab. 2] | | Limited GPU budget or no CrowdHuman-scale pretraining | Not TrackFormer as the paper trains it | Two days on seven GPUs, and the ablation removes CrowdHuman pretraining for a 2.0 MOTA loss [arxiv:2101.02702 §4.2] [arxiv:2101.02702 tab. 3] | | Accuracy matters more than speed on pedestrians | Not the RNN tracker | Its compared method MDP reports MOTA 30.3 at 1.1 FPS on the same test set [arxiv:1604.03635 tab. 2] | | No labelled tracking video for training | RNN tracker's training recipe, with the accuracy gap in mind | Trained only on synthetic trajectories; it reports results on real video from that model [arxiv:1604.03635 § Training Data] | | Counting people in groups from video, no accuracy target | DeepSORT pipeline as a blueprint only | No tracking metric or grouping accuracy is reported, and the camera calibration is missing [arxiv:2005.01385 §VI] | | Need MOT speed on SeaDronesSee data | Not supported by this page | No MOT fps is reported for SeaDronesSee [arxiv:2105.01922 §5.3] | | Pedestrians with private detections, a strong tracking-by-detection baseline, and appearance cues are available | StrongSORT (online row) or StrongSORT++ with AFLink and GSI | 63.5 HOTA and 78.5 IDF1 online, and 64.4 HOTA and 79.5 IDF1 for StrongSORT++, on MOT17 test [arxiv:2202.13514 tab. IV]. StrongSORT++ runs at 7.1 FPS against 7.5 for the online row [arxiv:2202.13514 tab. IV] | | An existing tracker fragments identities or leaves gaps, and appearance features are not wanted | AFLink (appearance-free) first; add GSI only if smoothing the observed boxes is acceptable | AFLink alone reaches 80.0 IDF1 against 79.5 for StrongSORTv1 on MOT17 validation [arxiv:2202.13514 tab. II]. GSI rewrites every box, so it changes observed boxes too [arxiv:2202.13514 §IV-B] [arxiv:2202.13514 tab. II] | | Association is the bottleneck on MOT20-like crowds, and missed detections are the larger cost | Not StrongSORT++ alone | On MOT20 test, StrongSORT++ has 770 IDs against 1,120 for ByteTrack, but 117,920 false negatives against 87,594 [arxiv:2202.13514 tab. V] [arxiv:2202.13514 §V-C] | ## Implementation sketch The SeaDronesSee card describes evaluation, not a tracker. The shape below is the flow the paper uses to score any submission: the test labels stay on the server, and the scores come from comparing predictions with them. The exact protocol for each metric is in the paper's cited sections, not restated here. ```pseudocode # Score tracker predictions for one clip set [arxiv:2105.01922 §3.3] [§5] # Input: predicted boxes and identities per frame, per clip # Output: task metrics def score_submission(predictions, clips, task): # Test labels are withheld; predictions are uploaded for scoring [§3.3] if clips.labels_are_withheld: return upload_to_server(predictions, task) # SOT: success and precision curves per clip [§5.2] if task == "SOT": return success_and_precision(predictions, clips.labels) # MOT: MOT benchmark protocol on the chosen object set [§4.3] [§5.3] return mot_protocol(predictions, clips.labels, task.object_set) ``` The RNN tracker follows a different shape, one step per frame. The steps are summarised here, with the paper's section for each [arxiv:1604.03635 § Our Approach]. The paper has no algorithm box, so this is a reconstruction. ```pseudocode # One online tracking step per frame [arxiv:1604.03635 § Our Approach] # Input: target states, existence probabilities, detections, hidden state # Output: updated states, existence (kept targets only), hidden state def track_step(states, existence, detections, hidden): # Predict each target's next state from its state and the RNN memory predicted, hidden = motion_rnn(states, hidden) # Distance from each predicted state to each detection cost = pairwise_distance(predicted, detections) # LSTM assigns detections to targets, one target row at a time; # the extra column means "no detection" assignment = lstm_associate(cost) # Weighted update of each target from its assigned evidence states = weighted_update(predicted, detections, assignment) # Existence for the next frame; targets at or below 0.6 are dropped existence = existence_head(states, existence, detections, assignment) keep = existence > 0.6 return states[keep], existence[keep], hidden ``` The TrackFormer paper's loop, one frame at a time, has the same shape as the RNN tracker's, but identity is carried by the query and not by an assignment step [arxiv:2101.02702 §3.2] [arxiv:2101.02702 app. A.4]. The sketch below omits the re-identification pool. ```pseudocode # Tracking by attention, one frame at a time [arxiv:2101.02702 §3.2] # Input: video frames; score threshold 0.4; NMS IoU 0.9 # Output: per-frame boxes with track identities def track_video(frames): tracks = [] for frame in frames: # Decode previous track embeddings with the learned object queries outputs = detect(frame, track_queries=[t.embedding for t in tracks]) # Keep a track only while its score stays above the threshold tracks = [t for t in tracks if outputs.score(t) > 0.4] # Start a track for each object query above the threshold tracks += new_tracks(outputs.object_queries, threshold=0.4) # Remove duplicate tracks by NMS at IoU 0.9 tracks = nms_tracks(tracks, iou_threshold=0.9) emit(tracks) ``` ## Contradictions Within the SeaDronesSee paper, the text and the table differ on one point: the text gives +6.4 AP50avg for the medium angle domain, and [tab. 9] gives 39.3 − 32.7 = 6.6 [arxiv:2105.01922 §5.4] [arxiv:2105.01922 tab. 9]. The paper also gives two altitude ranges for different rows of [tab. 1]: 5–260 m for detection, and 5–150 m for the single- and multi-object tracking rows [arxiv:2105.01922 tab. 1]. The paper does not say which clips reach which heights, so the tracking range cannot be confirmed from the card. Within the RNN paper, the abstract gives a higher speed than the table: about 300 Hz in the abstract [arxiv:1604.03635 § Abstract], and 165.2 FPS for the same method in [tab. 2] [arxiv:1604.03635 tab. 2]. Use the table value, and note that the hardware is not stated. Its test MOTA is 19.0 against 22.3 on the training set, and the paper does not discuss the gap [arxiv:1604.03635 tab. 1] [arxiv:1604.03635 tab. 2]. Within the SportsMOT paper, the same MixSort-Byte run is reported with two slightly different scores: MOTA 96.2 and 2,472 IDs in [tab. 3], against MOTA 96.1 and 2,469 IDs in [tab. 6] and [tab. 7]. The abstract's dataset ratios also differ from the table: the abstract says about 3 times the boxes of MOT17, but [tab. 1] gives about 5.6 times [arxiv:2304.05170 tab. 1]. The main-table fusion weight is not stated, and the released code's default of 0.6 is not one of the per-sport optima [arxiv:2304.05170 tab. 4]. Within the BAT paper, the ablation puts BAT-Dual at 56.4 SR on LasHeR, above the full BAT model at 56.3 [arxiv:2312.10611 tab. 2]. The number of LasHeR attributes is also given two ways: 19 in total with 7 new in §4.1.1, and 19 additional attributes in §4.3.3 [arxiv:2312.10611 §4.1.1] [arxiv:2312.10611 §4.3.3]. Check the dataset before citing either. Within the social-distancing paper, the detector used with DeepSORT is named differently in two places: Faster R-CNN in §III-E and YOLO v3 in §IV. The final choice is YOLO v3 [arxiv:2005.01385 §V]. The backbone for the detectors is named as Inception v2 in §V and as Darknet-53 for YOLO v3 in §III-D [arxiv:2005.01385 §III-D] [arxiv:2005.01385 §V]. Within the TrackFormer paper, the MOTS20 mask-training table does not match the text. The text says that joint mask training improves IDF1 by 1.2. The two IDF1 values in the table differ by 1.2, but the table's marker sits on the row with the higher value (56.0), so the table does not show which row is the joint-training run [arxiv:2101.02702 §4.4] [arxiv:2101.02702 tab. 4]. Treat the row labels as unresolved. The paper's text also describes its false-positive and false-negative augmentations per query [arxiv:2101.02702 §3.3], while the authors' released code draws batch-wide counts. The paper card's "Paper vs code" section lists these differences, so check which one you reproduce. Within the StrongSORT paper and its released code, several settings differ. The paper sets tracker-level NMS to 0.8 [arxiv:2202.13514 §V-A], but the released tracker uses 1.0, which suppresses nothing; 0.8 is used at detection export [arxiv:2202.13514 §V-A] [code: opts.py#L127-L128]. The paper gives a matching distance of 0.45, while the code uses 0.4 for BoT and adds 0.05 when the motion cost is on [arxiv:2202.13514 §V-A] [code: opts.py#L130-L137]. The paper writes the NSA noise as (1 − c)R, but the code scales the standard deviation, which squares the factor in the covariance [arxiv:2202.13514 eq. 9] [code: deep_sort/kalman_filter.py#L148-L151]. The paper's AFLink loss also differs from the code's two-class cross-entropy [arxiv:2202.13514 eq. 11] [code: AFLink/train.py#L34]. Check the paper and the code separately before you reproduce either. Within the SimTrack paper, Sim-B/16 is given 25.0 GFLOPs in [tab. 1] and 23.4 GFLOPs in [tab. 4] for the same joint model. The card does not resolve which is right [arxiv:2203.05328 tab. 1] [arxiv:2203.05328 tab. 4]. The text calls the GOT-10k score AUC, while the table labels it AO, and the text's 69.8 for Sim-L/14 is the table's AO value. Use AO [arxiv:2203.05328 §4.2] [arxiv:2203.05328 tab. 3]. Across the papers, the numbers are not comparable. They use different datasets, different object classes, different metrics (for example MPR and MSR on RGBT234 against PR and SR on LasHeR) and different hardware for speed. ## Papers - [SeaDronesSee: A Maritime Benchmark for Detecting Humans in Open Water](/papers/2105.01922): a UAV benchmark with SOT and MOT tasks, withheld test labels and author-reported tracker baselines. - [Online Multi-Target Tracking Using Recurrent Neural Networks](/papers/1604.03635): an online pedestrian tracker in which RNNs predict and update targets and an LSTM handles association, trained on synthetic trajectories. - [StrongSORT: Make DeepSORT Great Again](/papers/2202.13514): an upgraded DeepSORT with a stronger detector, appearance model and global matching, plus AFLink and GSI as optional post-processing. - [SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes](/papers/2304.05170): a sports tracking dataset with only on-court players tracked, and MixSort, an appearance-fusion plug-in for tracking-by-detection trackers. - [TrackFormer: Multi-Object Tracking with Transformers](/papers/2101.02702): an end-to-end transformer tracker in which track queries carry each object's identity from frame to frame, with a mask head for MOTS20. - [Bi-directional Adapter for Multi-modal Tracking](/papers/2312.10611): an RGB-thermal single-object tracker in which small bi-directional adapters let two streams prompt each other in a frozen ViT. - [Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking](/papers/2203.05328): a single-object tracker in which one ViT takes exemplar and search tokens together, with no Siamese correlation or transformer head. - [Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques](/papers/2005.01385): a people detector with DeepSORT and a distance-based grouping rule for surveillance video; the paper reports no tracking metric.