---
kind: concept
slug: tracking-by-detection
title: Tracking by detection
summary: >-
  Multi-object tracking by detecting objects in every frame, then linking detections into identities. Covers a DeepSORT-style baseline, StrongSORT with offline linking, POI's pedestrian tracker, a 3D tracker on fused two-vehicle LiDAR, a sports tracker with visual similarity, and a clip-level transformer without linking. Trade-offs: appearance against motion, online against offline.
tags: [multi-object-tracking, data-association, tracking-by-detection, appearance-features, interpolation, sports, cooperative-perception, 3d-tracking]
papers: ["2202.13514", "2304.05170", "2011.14503", "1610.06136", "2303.07601"]
related: [visual-object-tracking, object-detection, bipartite-matching, set-prediction, object-queries, gaussian-process, instance-segmentation, metric-learning, collaborative-perception, domain-adaptation]
status: draft
updated_at: 2026-11-01
---

## Overview

Tracking by detection runs an object detector on every frame and then links the
detections over time into tracks, one identity per object. The linking step,
called association, combines a motion prediction (usually a Kalman filter), an
appearance similarity (a re-identification feature per detection), or both. The
usual solver is the Hungarian algorithm on a cost matrix
[arxiv:2202.13514 §III-A] [arxiv:2304.05170 §4.1].

The authors of StrongSORT argue that the paradigm itself is sound, and that the
older DeepSORT tracker underperforms because its components are out of date
[arxiv:2202.13514 §I]. The authors of POI make a related point: detection
quality and a learned appearance feature are the main levers, and most prior
work focuses on the linking step [arxiv:1610.06136 §1]. The authors of VisTR
take a different view. They say that top-down methods, which detect instances
and then link them with hand-designed rules, depend heavily on per-frame
quality, and they replace the linking with a sequence-level prediction
[arxiv:2011.14503 §1]. The variants below cover these positions.

Five paper cards are relevant. StrongSORT, POI and V2V4Real list this slug in
their `concepts:`. SportsMOT and VisTR discuss tracking by detection in their
text, so they are linked here, but their `concepts:` lists do not include this
slug. The variants are as much a map of the design space as a survey.

## Variants

### Two-stage baseline: DeepSORT-style association

Each track keeps a bank of appearance features from its last 100 frames. The
appearance cost of a detection is the smallest cosine distance to that bank. A
Kalman filter predicts each track's box, and squared Mahalanobis distance gates
unlikely pairs. Association runs a matching cascade that favours recently seen
tracks, and each stage is solved with the Hungarian algorithm
[arxiv:2202.13514 §III-A] [arxiv:2202.13514 eq. 1] [arxiv:2202.13514 eq. 2].

Reference numbers, from the StrongSORT paper's own runs on MOT17 validation:
77.3 IDF1, 76.7 MOTA and 66.3 HOTA at 13.8 FPS [arxiv:2202.13514 tab. I]. The
same paper reports the baseline on MOT17 test at 61.2 HOTA and 74.5 IDF1
[arxiv:2202.13514 tab. IV]. This is the reference point for every upgrade
below. Card: [StrongSORT: Make DeepSORT Great Again](/papers/2202.13514).

### Upgraded tracker: StrongSORT

StrongSORT changes the parts of the baseline and adds no new algorithm to the
association step itself [arxiv:2202.13514 §III-B]. The changes are:

- a stronger detector (YOLOX-X) in place of Faster R-CNN;
- a stronger appearance model (BoT features);
- a smoothed appearance vector per track (an exponential moving average with
  factor 0.9) in place of the feature bank;
- camera-motion compensation with an ECC warp, applied before prediction;
- measurement noise in the Kalman update scaled down for low-confidence
  detections;
- a cost that mixes appearance and motion, with weight 0.98 on appearance;
- one global assignment in place of the matching cascade.

Each step was added on top of the one before in the authors' ablation
[arxiv:2202.13514 tab. I]. The text reports the gains as: BoT +2.2 IDF1, ECC
+0.2 IDF1 and +0.3 MOTA, noise scaling +0.4 HOTA, the moving average +0.4 IDF1,
and the motion cost +0.8 IDF1 [arxiv:2202.13514 §V-B]. The largest IDF1 gain comes
from BoT, and it costs speed: frames per second fall from 13.8 to 8.3 when BoT
is added [arxiv:2202.13514 tab. I]. The global assignment helps
only when the tracker is already strong. It gives +1.4 IDF1 on the strongest
StrongSORT row but −1.1 IDF1 on DeepSORT [arxiv:2202.13514 tab. III].

On MOT17 test, under the private-detection protocol, StrongSORT reaches
63.5 HOTA and 78.5 IDF1 as an online tracker [arxiv:2202.13514 tab. IV].
The authors report about 7.5 FPS on a single V100 [arxiv:2202.13514 §V-A].

### Post-hoc linking and interpolation: AFLink and GSI

These two steps run after tracking, on the finished output, so they are offline.
They are optional and can be attached to other trackers
[arxiv:2202.13514 §IV].

- **AFLink** links tracklets that belong to the same object. It uses only
  frame numbers and positions, not appearance. It takes the last 30 frames of
  one tracklet and the first 30 of the next, keeps only pairs that start 0 to
  29 frames apart and within 75 pixels, scores them with a small network, and
  solves a one-to-one assignment, keeping links that score at least 0.95
  [arxiv:2202.13514 §IV-A] [arxiv:2202.13514 §V-A]. The AFLink training data
  appears to come from the sequences used for its ablation validation, so the
  ablation gains may be optimistic; the MOT17 test numbers do not share this
  issue [arxiv:2202.13514 § Paper vs code].
- **GSI** replaces straight-line gap filling with Gaussian-process smoothing of
  each coordinate, with a length scale that depends on track length. It fills
  gaps shorter than 20 frames [arxiv:2202.13514 §IV-B] [arxiv:2202.13514 §V-A].
  It rewrites every box of a track, not only the filled ones
  [arxiv:2202.13514 §IV-B].

On MOT17 validation, adding both steps raised HOTA by 1.1 to 2.3 across six
trackers: StrongSORT variants, CenterTrack, TransTrack and FairMOT
[arxiv:2202.13514 tab. II]. The largest gain, +2.3 HOTA, is for CenterTrack, and
it costs 1.6 FPS. The AFLink windows are pixel and frame values tied to MOT17's
frame rate and resolution, so they need recalibration for other data
[arxiv:2202.13514 § Paper vs code]. The StrongSORT++ MOT17 test result combines
StrongSORT with both steps: 64.4 HOTA and 79.5 IDF1 [arxiv:2202.13514 tab. IV].

### Stronger detector and embedding, two-stage matching: POI

POI keeps the paradigm and puts most of its effort into the detector and the
appearance feature. The detector is a Faster R-CNN, and the feature is a
128-d embedding trained with a softmax loss and a triplet loss
[arxiv:1610.06136 §1] [arxiv:1610.06136 §2]. The online tracker predicts each
track with a Kalman filter and matches with the Kuhn-Munkres algorithm. The
matching has two stages in the text: high-quality tracklets against all
detections first, then the remaining tracklets against the rest. Algorithm 1
draws it as one matching [arxiv:1610.06136 §3] [arxiv:1610.06136 alg. 1]. The
affinity is the product of a cosine appearance term, a centre-motion term and a
shape term [arxiv:1610.06136 eq. 4]. A track that is lost for more than 100
frames is dropped [arxiv:1610.06136 §5].

The offline tracker is an improved H2T. It cuts the video into segments, links
detections inside each one, and merges neighbouring segments until one remains
[arxiv:1610.06136 alg. 2]. For targets of very different sizes, it adds scale
gating [arxiv:1610.06136 §4].

On the MOT16 rank list, the paper reports 66.1 MOTA for the online tracker and
68.2 for the offline one [arxiv:1610.06136 tab. 4]. The ablations use the MOT16
train set. With the same detector, the learned feature gives 65.2 MOTA against
63.6 for a GoogLeNet-style feature [arxiv:1610.06136 tab. 2]. The detector
threshold is tuned on that train set, and the paper uses a different value for
two of the sequences [arxiv:1610.06136 §5]. There is no official code. The
abstract links a folder that the authors describe as holding their detections
and appearance features, which the card has not inspected. The learning rates, the triplet margin and the Kalman noise values are not stated,
so an implementer must choose them for the embedding [arxiv:1610.06136 §2]
[arxiv:1610.06136 §3]. Card:
[POI: Multiple Object Tracking with High Performance Detection and Appearance Feature](/papers/1610.06136).

### Motion plus visual similarity: MixSort

SportsMOT introduces a dataset of basketball, volleyball and football games, in
which only the players on the court are labelled. Its tracker, MixSort, plugs
into ByteTrack or OC-SORT. For each existing track, a Kalman prediction gives a
search region. A MixFormer-style network, given the track's template crop,
produces a heatmap over that region. The heatmap value at each detection's
centre becomes a visual similarity. That value is blended with IoU, with weight
α on IoU, and the result is solved with the Hungarian algorithm
[arxiv:2304.05170 §4.1] [arxiv:2304.05170 §4.2] [arxiv:2304.05170 eq. 1].

On the SportsMOT test set, trained on train plus validation, the paper reports
MixSort-Byte at 65.7 HOTA against 64.1 for ByteTrack, and MixSort-OC at 74.1
against 73.7 for OC-SORT [arxiv:2304.05170 tab. 3]. These gains are small, 0.4
to 1.6 HOTA [arxiv:2304.05170 tab. 3]. The best α differs per sport: 0.8
for basketball, 0.5 for volleyball and 0.4 for football [arxiv:2304.05170 tab. 4].
The template is replaced only when a detection passes a test on its uncovered
area, and the threshold is not stated in the paper [arxiv:2304.05170 §4.3].
Most of the MixSort-Byte gain over ByteTrack is in AssA and IDF1, while DetA
is nearly the same (78.8 against 78.5) [arxiv:2304.05170 tab. 3]. The authors
name association as the main difficulty on this dataset [arxiv:2304.05170 §1].

### Clip-level sequence prediction, no linking: VisTR

VisTR does not link per-frame detections. It takes a clip of 36 frames and
predicts 10 instance sequences in one pass. Slot k in every frame is the same
instance, so identity is fixed by construction. Hungarian matching then
assigns predicted sequences to ground-truth sequences as a whole
[arxiv:2011.14503 §3.1] [arxiv:2011.14503 §3.2].

This is video instance segmentation, not multi-object tracking, so its numbers
are not comparable with the MOT tables above. On YouTube-VIS validation with
ResNet-101 it reports 40.1 AP at 27.7 FPS with data loading, and 57.7 FPS
without [arxiv:2011.14503 tab. 2]. Its limits are structural. Clips are fixed at
36 frames, and the released inference code reads only the first 36 frames of a
video [arxiv:2011.14503 § Paper vs code]. Card:
[End-to-End Video Instance Segmentation with Transformers](/papers/2011.14503).

### Cooperative 3D tracking on fused detections: V2V4Real

V2V4Real tracks objects in 3D from two vehicles' LiDAR. Its tracker is
AB3DMOT, and its detections come from the cooperative detector
[arxiv:2303.07601 §4.2] [arxiv:2303.07601 §5.1]. Per frame, each track is moved
ahead at constant velocity, a 3D IoU between predictions and detections gives
the affinity, and the Hungarian algorithm pairs them. A matched track gets a
Kalman update. A new track starts only from detections that persist over the
next few frames, and a track is removed after several frames without a match
[arxiv:2303.07601 app. C.2]. The paper does not state the IoU gate, the Kalman
noise values, or the exact frame counts for birth and death
[arxiv:2303.07601 app. C.2].

On the paper's cooperative tracking benchmark, AMOTA is 16.08 with no fusion,
29.28 with late fusion, and 32.12 with CoBEVT [arxiv:2303.07601 tab. 4]. The
rows differ in the fusion method that produces the detections
[arxiv:2303.07601 §4.2]. The benchmarks use LiDAR only, and the camera images and maps
are not used in any task [arxiv:2303.07601 §4.3]. Card:
[V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception](/papers/2303.07601).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Online tracking, a detector and an appearance model are available | Two-stage baseline or StrongSORT, online mode | The online StrongSORT row reports 63.5 HOTA on MOT17 test [arxiv:2202.13514 tab. IV]; BoT appearance features give the largest IDF1 gain in its ablation [arxiv:2202.13514 §V-B] |
| Latency matters more than the last points of HOTA | Two-stage baseline with fewer add-ons | BoT features cut FPS from 13.8 to 8.3 in the authors' ablation [arxiv:2202.13514 tab. I] |
| Offline output is acceptable and association quality is the goal | StrongSORT++ (AFLink and GSI on top) | Offline rows give the best MOT17 test HOTA [arxiv:2202.13514 tab. IV] |
| Pedestrians, and you can train an appearance embedding yourself | POI, online or offline | Two-stage matching with a product affinity [arxiv:1610.06136 §3]; 66.1 MOTA online on the MOT16 rank list [arxiv:1610.06136 tab. 4]. The training recipe is not stated, so the embedding is your work [arxiv:1610.06136 §2] |
| No appearance model, only boxes and frame numbers | AFLink on any tracker | Its input is positions and timing, not appearance [arxiv:2202.13514 §IV-A]; recalibrate its windows for new data |
| Frequent missed detections causing gaps | GSI after tracking | It smooths and fills gaps under a 20-frame limit [arxiv:2202.13514 §V-A]; it costs some speed (1.6 FPS for CenterTrack) [arxiv:2202.13514 tab. II] |
| Fast, similar-looking players in sports video | MixSort on ByteTrack or OC-SORT, with α tuned per sport | Visual similarity is blended with IoU, and the best α differs by sport [arxiv:2304.05170 tab. 4] |
| Dance-like data where IoU already works well (DanceTrack) | Check before adopting MixSort | On DanceTrack validation MixSort-Byte lowers HOTA slightly, although it raises IDF1 [arxiv:2304.05170 tab. 8] |
| Fixed short clips where the output needs masks, and linking is not wanted | Clip-level sequence prediction (VisTR-style) | Identity is fixed by slot order, with no linking step [arxiv:2011.14503 §3.1] |
| Several vehicles share detections, and 3D tracks are needed from LiDAR | Tracking by detection on fused detections, AB3DMOT-style (V2V4Real) | AMOTA rises from 16.08 with no fusion to 29.28 with late fusion and 32.12 with CoBEVT [arxiv:2303.07601 tab. 4]. The tracker's IoU gate and noise values are not stated, so tune them yourself [arxiv:2303.07601 app. C.2]. The dataset's terms do not grant reuse, so check them first ([card](/papers/2303.07601)) |

## Implementation sketch

Per-frame loop shared by the variants above. The comments mark where the
variants differ. The thresholds are the StrongSORT paper's values
[arxiv:2202.13514 §V-A].

```pseudocode
# One frame of tracking by detection  [arxiv:2202.13514 §III]
# Input:  detections (box, score, feature), tracks, camera warp
# Output: updated tracks

def track_step(detections, tracks, warp):
    # 1. Drop low-confidence detections (0.6 in StrongSORT) [§V-A]
    detections = [d for d in detections if d.score >= 0.6]
    # 2. Camera-motion compensation  [§III-B]
    for track in tracks:
        track.warp_box(warp)
    # 3. Kalman prediction, one step ahead  [§III-A]
    for track in tracks:
        track.predict()
    # 4. Variant point: appearance-only (DeepSORT), mixed (StrongSORT),
    #    or IoU plus visual similarity (MixSort, alpha weight)  [eq. 10]
    cost = mixed_cost(tracks, detections, lam=0.98)
    # 5. Variant point: matching cascade (DeepSORT), one global
    #    assignment (StrongSORT), or two stages, high-quality tracklets
    #    first (POI), rejecting costs above 0.45  [§III-B] [POI §3]
    matches = assign(cost, max_distance=0.45)
    # 6. Kalman update; noise scaled by detection confidence  [eq. 9]
    for track, det in matches:
        track.kalman_update(det, noise_scale=1 - det.score)
    # 7. Exponential moving average of the appearance vector  [eq. 7]
    for track, det in matches:
        track.ema_update(det.feature, alpha=0.9)
    return tracks


# Offline, after the whole sequence is tracked  [§IV]
# 8. Optional: link tracklets, then interpolate gaps  [§IV-A] [§IV-B]
def post_process(output):
    output = aflink_link(output)      # appearance-free; 75 px, 30 frames
    return gsi_interpolate(output)    # gaps under 20 frames
```

Notes on the sketch. The paper writes the noise scaling as a factor on the
noise matrix, but the released code scales the standard deviation, which
scales the variance by the square of the factor [arxiv:2202.13514 § Paper vs
code]. The paper does not state the
order of camera compensation and prediction, but the code applies the warp
first [arxiv:2202.13514 § Paper vs code]. The sketch does not cover the
visual-similarity term of MixSort, which needs a second network per track
[arxiv:2304.05170 §4.1].

## Contradictions

- **Motion prediction helps on MOT17 but not on SportsMOT.** StrongSORT reports
  that adding the motion cost gives +0.8 IDF1 on MOT17 validation
  [arxiv:2202.13514 §V-B]. On SportsMOT test, the authors' ablation with IoU
  plus Kalman prediction, against IoU alone, changes IDF1 only from 71.2 to
  71.4. It lowers HOTA from 71.5 to 64.1 and AssA from 58.1 to 52.3, and the
  paper concludes the Kalman filter hurts there [arxiv:2304.05170 tab. 6]
  [arxiv:2304.05170 §5.3]. The datasets, detectors and motion settings differ,
  and neither paper runs the other's setting, so this is not resolved here.
- **Visual similarity helps HOTA on the test sets but not on DanceTrack
  validation.** MixSort-Byte gains HOTA on SportsMOT test and MOT17 test
  [arxiv:2304.05170 tab. 3] [arxiv:2304.05170 tab. 5]. On DanceTrack validation
  it lowers HOTA from 47.1 to 46.7 while raising IDF1 from 51.9 to 53.0
  [arxiv:2304.05170 tab. 8].
- **Is tracking by detection the right paradigm?** StrongSORT argues the
  paradigm is sound and the components are outdated [arxiv:2202.13514 §I].
  VisTR argues that top-down linking is slow and relies on hand-designed rules,
  and it replaces linking with sequence prediction [arxiv:2011.14503 §1]. The
  two papers test different tasks (MOT and video instance segmentation), so
  neither result settles the question.

## Papers

- [StrongSORT: Make DeepSORT Great Again](/papers/2202.13514): upgrades DeepSORT
  with a stronger detector, appearance model and association, and adds AFLink
  and GSI as offline steps.
- [POI: Multiple Object Tracking with High Performance Detection and Appearance Feature](/papers/1610.06136):
  an online and an offline pedestrian tracker built on a Faster R-CNN detector
  and a learned 128-d appearance embedding.
- [SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes](/papers/2304.05170):
  a sports dataset and MixSort, which blends a visual-similarity term into
  ByteTrack and OC-SORT.
- [End-to-End Video Instance Segmentation with Transformers](/papers/2011.14503):
  a clip-level transformer that predicts instance sequences without a linking
  step, as a contrast to tracking by detection.
- [V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception](/papers/2303.07601):
  a real two-vehicle LiDAR dataset and benchmark, with an AB3DMOT-based 3D
  tracker on fused detections.