Tracking by detection

Overview

Tracking by detection runs an object detector on every frame and then links the detections over time into tracks, one identity per object. The linking step, called association, combines a motion prediction (usually a Kalman filter), an appearance similarity (a re-identification feature per detection), or both. The usual solver is the Hungarian algorithm on a cost matrix 2202.13514 §III-A 2304.05170 §4.1.

The authors of StrongSORT argue that the paradigm itself is sound, and that the older DeepSORT tracker underperforms because its components are out of date 2202.13514 §I. The authors of POI make a related point: detection quality and a learned appearance feature are the main levers, and most prior work focuses on the linking step 1610.06136 §1. The authors of VisTR take a different view. They say that top-down methods, which detect instances and then link them with hand-designed rules, depend heavily on per-frame quality, and they replace the linking with a sequence-level prediction 2011.14503 §1. The variants below cover these positions.

Five paper cards are relevant. StrongSORT, POI and V2V4Real list this slug in their concepts:. SportsMOT and VisTR discuss tracking by detection in their text, so they are linked here, but their concepts: lists do not include this slug. The variants are as much a map of the design space as a survey.

Variants

Two-stage baseline: DeepSORT-style association

Each track keeps a bank of appearance features from its last 100 frames. The appearance cost of a detection is the smallest cosine distance to that bank. A Kalman filter predicts each track's box, and squared Mahalanobis distance gates unlikely pairs. Association runs a matching cascade that favours recently seen tracks, and each stage is solved with the Hungarian algorithm 2202.13514 §III-A 2202.13514 eq. 1 2202.13514 eq. 2.

Reference numbers, from the StrongSORT paper's own runs on MOT17 validation: 77.3 IDF1, 76.7 MOTA and 66.3 HOTA at 13.8 FPS 2202.13514 tab. I. The same paper reports the baseline on MOT17 test at 61.2 HOTA and 74.5 IDF1 2202.13514 tab. IV. This is the reference point for every upgrade below. Card: StrongSORT: Make DeepSORT Great Again.

Upgraded tracker: StrongSORT

StrongSORT changes the parts of the baseline and adds no new algorithm to the association step itself 2202.13514 §III-B. The changes are:

• a stronger detector (YOLOX-X) in place of Faster R-CNN;
• a stronger appearance model (BoT features);
• a smoothed appearance vector per track (an exponential moving average with factor 0.9) in place of the feature bank;
• camera-motion compensation with an ECC warp, applied before prediction;
• measurement noise in the Kalman update scaled down for low-confidence detections;
• a cost that mixes appearance and motion, with weight 0.98 on appearance;
• one global assignment in place of the matching cascade.

Each step was added on top of the one before in the authors' ablation 2202.13514 tab. I. The text reports the gains as: BoT +2.2 IDF1, ECC +0.2 IDF1 and +0.3 MOTA, noise scaling +0.4 HOTA, the moving average +0.4 IDF1, and the motion cost +0.8 IDF1 2202.13514 §V-B. The largest IDF1 gain comes from BoT, and it costs speed: frames per second fall from 13.8 to 8.3 when BoT is added 2202.13514 tab. I. The global assignment helps only when the tracker is already strong. It gives +1.4 IDF1 on the strongest StrongSORT row but −1.1 IDF1 on DeepSORT 2202.13514 tab. III.

On MOT17 test, under the private-detection protocol, StrongSORT reaches 63.5 HOTA and 78.5 IDF1 as an online tracker 2202.13514 tab. IV. The authors report about 7.5 FPS on a single V100 2202.13514 §V-A.

Post-hoc linking and interpolation: AFLink and GSI

These two steps run after tracking, on the finished output, so they are offline. They are optional and can be attached to other trackers 2202.13514 §IV.

• AFLink links tracklets that belong to the same object. It uses only frame numbers and positions, not appearance. It takes the last 30 frames of one tracklet and the first 30 of the next, keeps only pairs that start 0 to 29 frames apart and within 75 pixels, scores them with a small network, and solves a one-to-one assignment, keeping links that score at least 0.95 2202.13514 §IV-A 2202.13514 §V-A. The AFLink training data appears to come from the sequences used for its ablation validation, so the ablation gains may be optimistic; the MOT17 test numbers do not share this issue 2202.13514 § Paper vs code.
• GSI replaces straight-line gap filling with Gaussian-process smoothing of each coordinate, with a length scale that depends on track length. It fills gaps shorter than 20 frames 2202.13514 §IV-B 2202.13514 §V-A. It rewrites every box of a track, not only the filled ones 2202.13514 §IV-B.

On MOT17 validation, adding both steps raised HOTA by 1.1 to 2.3 across six trackers: StrongSORT variants, CenterTrack, TransTrack and FairMOT 2202.13514 tab. II. The largest gain, +2.3 HOTA, is for CenterTrack, and it costs 1.6 FPS. The AFLink windows are pixel and frame values tied to MOT17's frame rate and resolution, so they need recalibration for other data 2202.13514 § Paper vs code. The StrongSORT++ MOT17 test result combines StrongSORT with both steps: 64.4 HOTA and 79.5 IDF1 2202.13514 tab. IV.

Stronger detector and embedding, two-stage matching: POI

POI keeps the paradigm and puts most of its effort into the detector and the appearance feature. The detector is a Faster R-CNN, and the feature is a 128-d embedding trained with a softmax loss and a triplet loss 1610.06136 §1 1610.06136 §2. The online tracker predicts each track with a Kalman filter and matches with the Kuhn-Munkres algorithm. The matching has two stages in the text: high-quality tracklets against all detections first, then the remaining tracklets against the rest. Algorithm 1 draws it as one matching 1610.06136 §3 1610.06136 alg. 1. The affinity is the product of a cosine appearance term, a centre-motion term and a shape term 1610.06136 eq. 4. A track that is lost for more than 100 frames is dropped 1610.06136 §5.

The offline tracker is an improved H2T. It cuts the video into segments, links detections inside each one, and merges neighbouring segments until one remains 1610.06136 alg. 2. For targets of very different sizes, it adds scale gating 1610.06136 §4.

On the MOT16 rank list, the paper reports 66.1 MOTA for the online tracker and 68.2 for the offline one 1610.06136 tab. 4. The ablations use the MOT16 train set. With the same detector, the learned feature gives 65.2 MOTA against 63.6 for a GoogLeNet-style feature 1610.06136 tab. 2. The detector threshold is tuned on that train set, and the paper uses a different value for two of the sequences 1610.06136 §5. There is no official code. The abstract links a folder that the authors describe as holding their detections and appearance features, which the card has not inspected. The learning rates, the triplet margin and the Kalman noise values are not stated, so an implementer must choose them for the embedding 1610.06136 §2 1610.06136 §3. Card: POI: Multiple Object Tracking with High Performance Detection and Appearance Feature.

Motion plus visual similarity: MixSort

SportsMOT introduces a dataset of basketball, volleyball and football games, in which only the players on the court are labelled. Its tracker, MixSort, plugs into ByteTrack or OC-SORT. For each existing track, a Kalman prediction gives a search region. A MixFormer-style network, given the track's template crop, produces a heatmap over that region. The heatmap value at each detection's centre becomes a visual similarity. That value is blended with IoU, with weight α on IoU, and the result is solved with the Hungarian algorithm 2304.05170 §4.1 2304.05170 §4.2 2304.05170 eq. 1.

On the SportsMOT test set, trained on train plus validation, the paper reports MixSort-Byte at 65.7 HOTA against 64.1 for ByteTrack, and MixSort-OC at 74.1 against 73.7 for OC-SORT 2304.05170 tab. 3. These gains are small, 0.4 to 1.6 HOTA 2304.05170 tab. 3. The best α differs per sport: 0.8 for basketball, 0.5 for volleyball and 0.4 for football 2304.05170 tab. 4. The template is replaced only when a detection passes a test on its uncovered area, and the threshold is not stated in the paper 2304.05170 §4.3. Most of the MixSort-Byte gain over ByteTrack is in AssA and IDF1, while DetA is nearly the same (78.8 against 78.5) 2304.05170 tab. 3. The authors name association as the main difficulty on this dataset 2304.05170 §1.

Clip-level sequence prediction, no linking: VisTR

VisTR does not link per-frame detections. It takes a clip of 36 frames and predicts 10 instance sequences in one pass. Slot k in every frame is the same instance, so identity is fixed by construction. Hungarian matching then assigns predicted sequences to ground-truth sequences as a whole 2011.14503 §3.1 2011.14503 §3.2.

This is video instance segmentation, not multi-object tracking, so its numbers are not comparable with the MOT tables above. On YouTube-VIS validation with ResNet-101 it reports 40.1 AP at 27.7 FPS with data loading, and 57.7 FPS without 2011.14503 tab. 2. Its limits are structural. Clips are fixed at 36 frames, and the released inference code reads only the first 36 frames of a video 2011.14503 § Paper vs code. Card: End-to-End Video Instance Segmentation with Transformers.

Cooperative 3D tracking on fused detections: V2V4Real

V2V4Real tracks objects in 3D from two vehicles' LiDAR. Its tracker is AB3DMOT, and its detections come from the cooperative detector 2303.07601 §4.2 2303.07601 §5.1. Per frame, each track is moved ahead at constant velocity, a 3D IoU between predictions and detections gives the affinity, and the Hungarian algorithm pairs them. A matched track gets a Kalman update. A new track starts only from detections that persist over the next few frames, and a track is removed after several frames without a match 2303.07601 app. C.2. The paper does not state the IoU gate, the Kalman noise values, or the exact frame counts for birth and death 2303.07601 app. C.2.

On the paper's cooperative tracking benchmark, AMOTA is 16.08 with no fusion, 29.28 with late fusion, and 32.12 with CoBEVT 2303.07601 tab. 4. The rows differ in the fusion method that produces the detections 2303.07601 §4.2. The benchmarks use LiDAR only, and the camera images and maps are not used in any task 2303.07601 §4.3. Card: V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception.

When to use which

Implementation sketch

Per-frame loop shared by the variants above. The comments mark where the variants differ. The thresholds are the StrongSORT paper's values 2202.13514 §V-A.

# One frame of tracking by detection  [arxiv:2202.13514 §III]
# Input:  detections (box, score, feature), tracks, camera warp
# Output: updated tracks

def track_step(detections, tracks, warp):
    # 1. Drop low-confidence detections (0.6 in StrongSORT) [§V-A]
    detections = [d for d in detections if d.score >= 0.6]
    # 2. Camera-motion compensation  [§III-B]
    for track in tracks:
        track.warp_box(warp)
    # 3. Kalman prediction, one step ahead  [§III-A]
    for track in tracks:
        track.predict()
    # 4. Variant point: appearance-only (DeepSORT), mixed (StrongSORT),
    #    or IoU plus visual similarity (MixSort, alpha weight)  [eq. 10]
    cost = mixed_cost(tracks, detections, lam=0.98)
    # 5. Variant point: matching cascade (DeepSORT), one global
    #    assignment (StrongSORT), or two stages, high-quality tracklets
    #    first (POI), rejecting costs above 0.45  [§III-B] [POI §3]
    matches = assign(cost, max_distance=0.45)
    # 6. Kalman update; noise scaled by detection confidence  [eq. 9]
    for track, det in matches:
        track.kalman_update(det, noise_scale=1 - det.score)
    # 7. Exponential moving average of the appearance vector  [eq. 7]
    for track, det in matches:
        track.ema_update(det.feature, alpha=0.9)
    return tracks


# Offline, after the whole sequence is tracked  [§IV]
# 8. Optional: link tracklets, then interpolate gaps  [§IV-A] [§IV-B]
def post_process(output):
    output = aflink_link(output)      # appearance-free; 75 px, 30 frames
    return gsi_interpolate(output)    # gaps under 20 frames

Notes on the sketch. The paper writes the noise scaling as a factor on the noise matrix, but the released code scales the standard deviation, which scales the variance by the square of the factor [arxiv:2202.13514 § Paper vs code]. The paper does not state the order of camera compensation and prediction, but the code applies the warp first 2202.13514 § Paper vs code. The sketch does not cover the visual-similarity term of MixSort, which needs a second network per track 2304.05170 §4.1.

Contradictions

• Motion prediction helps on MOT17 but not on SportsMOT. StrongSORT reports that adding the motion cost gives +0.8 IDF1 on MOT17 validation 2202.13514 §V-B. On SportsMOT test, the authors' ablation with IoU plus Kalman prediction, against IoU alone, changes IDF1 only from 71.2 to 71.4. It lowers HOTA from 71.5 to 64.1 and AssA from 58.1 to 52.3, and the paper concludes the Kalman filter hurts there 2304.05170 tab. 6 2304.05170 §5.3. The datasets, detectors and motion settings differ, and neither paper runs the other's setting, so this is not resolved here.
• Visual similarity helps HOTA on the test sets but not on DanceTrack validation. MixSort-Byte gains HOTA on SportsMOT test and MOT17 test 2304.05170 tab. 3 2304.05170 tab. 5. On DanceTrack validation it lowers HOTA from 47.1 to 46.7 while raising IDF1 from 51.9 to 53.0 2304.05170 tab. 8.
• Is tracking by detection the right paradigm? StrongSORT argues the paradigm is sound and the components are outdated 2202.13514 §I. VisTR argues that top-down linking is slow and relies on hand-designed rules, and it replaces linking with sequence prediction 2011.14503 §1. The two papers test different tasks (MOT and video instance segmentation), so neither result settles the question.

Papers

• StrongSORT: Make DeepSORT Great Again: upgrades DeepSORT with a stronger detector, appearance model and association, and adds AFLink and GSI as offline steps.
• POI: Multiple Object Tracking with High Performance Detection and Appearance Feature: an online and an offline pedestrian tracker built on a Faster R-CNN detector and a learned 128-d appearance embedding.
• SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes: a sports dataset and MixSort, which blends a visual-similarity term into ByteTrack and OC-SORT.
• End-to-End Video Instance Segmentation with Transformers: a clip-level transformer that predicts instance sequences without a linking step, as a contrast to tracking by detection.
• V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception: a real two-vehicle LiDAR dataset and benchmark, with an AB3DMOT-based 3D tracker on fused detections.