wikillm

Paper · 2022-09-11 · cs.CV

In Pursuit of Many: A Review of Modern Multiple Object Tracking Systems

A survey of multiple object tracking. It groups methods by the challenges they target (occlusion, identity switches, speed, scale) and by paradigm (tracking-by-detection, transformers, motion, graph, attention, foundation models, diffusion, Mamba). It reviews benchmarks and metrics (MOTA, IDF1, HOTA, AMOTA, motion-centric). It proposes no new method and runs no experiments.

Visual object trackingTracking by detectionObject queries (learned query slots in transformer decoders)multi-object-trackingsurveyevaluation-metricsbenchmarks

arXivPDFMarkdown8 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Multiple object tracking means following several things in a video at once and giving each one a stable label, so the same car or person keeps its identity from frame to frame. It is hard because objects hide behind each other, crowds are dense, detectors miss things, and an object that leaves and re-enters the frame often gets a new label. This review does not add a tracker. It sorts the many existing answers by the idea they rely on: linking detections over time, learning the links with transformers, predicting motion, building graphs, using memory and attention, or reusing large pretrained models. It also explains the datasets and scores people use to compare trackers, and where those scores fall short. Its aim is a map from each problem to the design choices that address it.

TL;DR

  • A survey with no new method and no experiments. The source used is arXiv v2 (updated 2026-03-07) §1.
  • Methods are organised two ways: by the failure mode they target §2 and by the modelling paradigm they use, in ten groups §3.
  • It catalogues 25 public MOT datasets tab. 11 and defines the classic metrics (MOTA, IDF1, Track-mAP), HOTA with its sub-metrics, MOTSA, AMOTA and motion-centric metrics §5.
  • Use it as a map of the field. Several metric definitions in it are inconsistent with the text, so check them against the original metric papers before implementing (see Pitfalls).

Problem

Multiple object tracking has to recover persistent identities from noisy, partial and changing observations. Occlusion, poor detections, and limited compute make these failures interact §2. The authors argue that existing surveys are mostly organised around methodological splits and leaderboards. They say those surveys leave the mapping from concrete failure modes (long occlusion, dense crowds, irregular motion, domain shift) to design choices largely implicit, and treat deployment constraints and newer evaluation protocols only at a high level §1. Their stated gap-filling contributions are a broader review beyond deep-learning pipelines, a challenge-based organisation, a dataset and metric summary, and a discussion of application constraints §1.

Method

The paper has no method of its own. Its "method" is a taxonomy, and the authors say the grouping is meant to help readers choose an approach by design principle (what it assumes about detections, temporal signal and compute), not by implementation detail §3.

Survey scope. The authors state they synthesised "over a hundred papers" §1. The text gives no selection criteria, search databases, or time window (see Pitfalls).

Challenges (§2). Each theme defines the problem, lists its empirical symptoms, names broad mitigations and says why it remains open §2.

  • Occlusion §2.1. Symptoms: detection dropouts, fragmented tracks, re-initialisation and identity switches. Mitigations: longer temporal memory, joint segmentation and tracking, global graph or appearance reasoning, and post-detection processing that keeps low-score detections.
  • Detection noise and background complexity §2.2. Symptoms: false positives, missed detections, camera-motion cues mistaken for object motion. Mitigations: background modelling and motion compensation, contextual detectors, and keeping more candidates for association.
  • Detection speed and real-time constraints §2.3. Mitigations: lightweight backbones, efficient attention, compression and hardware acceleration.
  • Lighting and imaging degradation §2.4. Mitigations: preprocessing, extra sensors (infrared, depth), and low-light training.
  • Identity switching §2.5. Causes: similar appearance, abrupt motion, occlusion, unstable detectors. Mitigations: memory and re-identification buffers, multi-frame or global association, graph optimisation, verification and correction.
  • Scale variation §2.6. Mitigations: scale-aware and multi-scale detectors, dynamic scale estimation, augmentation.
  • Model size and deployment §2.7. Mitigations: compact backbones, distillation, sparse attention.
  • Miscellaneous §2.8. Motion blur, when to start and end tracks, non-rigid deformation and inter-class similarity.

Approach groups (§3). The authors order the groups roughly by community activity, and show the share of papers per group as a pie chart fig. 3. The chart's values are not given in the text.

GroupAssumption the survey statesSummary table
Detection and target association §3.1A per-frame detector supplies candidates; appearance and short-term motion resolve most frames; low-score candidates are kepttab. 1
Transformers §3.2Queries split into detection and track queries; cross-attention both localises objects and resolves correspondencetab. 2
Motion model §3.3Object dynamics and scene geometry give reliable association priors; dynamics are smooth or learnabletab. 3
Graph model §3.4Pairwise and higher-order affinities, passed by message passing, resolve ambiguities that local matching cannottab. 4
Attention module §3.5Attended foreground features and per-track memory slots carry identity cues through occlusiontab. 5
Foundation models §3.6Large-scale pretraining transfers to MOT with little adaptationtab. 6
Siamese network §3.7An embedding puts same-object pairs close and different-object pairs aparttab. 7
Tracklet association §3.8Short local associations are reliable; linking fragments gives long trajectoriestab. 8
Generative tracking §3.9Iteratively denoising noisy proposals handles ambiguity better than deterministic regressiontab. 9
Mamba-based tracking §3.10State-space kernels model long sequences in linear timetab. 10

The graph group is the only one with an explicit pipeline in the text §3.4. Written as pseudocode (our wording of the paper's description):

Pseudocode
# Graph-based association, as described in the survey  [§3.4]
# Input:  detections for each frame
# Output: an identity for each node

def graph_association(frames):
    # Nodes are detections or tracklets  [§3.4]
    nodes = build_nodes(frames)
    # Pairwise features: appearance, motion and geometry  [§3.4]
    edges = pairwise_features(nodes)
    # Message passing (for example a GNN) refines node and edge embeddings  [§3.4]
    nodes, edges = message_passing(nodes, edges)
    # Decode edges into links, then a global assignment or ranking  [§3.4]
    links = decode_edges(edges)
    return global_assignment(nodes, links)

Evaluation equations. The survey gives these definitions. Symbols are as the paper defines them; see Pitfalls for the ones that disagree with the text.

MOTP, the localisation term of the classic metrics §5.1.1:

MOTP = \frac{\sum_{i,t} d^i_t}{\sum_t c_t}

MOTA, which combines misses, false positives and identity switches §5.1.2:

MOTA = 1 - \frac{\sum_t (m_t + fp_t + mme_t)}{\sum_t g_t}

IDF1, from trajectory-level identity precision and recall §5.1.3:

IDF1 = \frac{|IDTP|}{|IDTP| + 0.5|IDFP| + 0.5|IDFN|}

HOTA at one localisation threshold α, and its per-detection association accuracy A(c) §5.2:

HOTA_\alpha = \sqrt{\frac{\sum_{c \in TP} A(c)}{|TP| + |FN| + |FP|}}

A(c) = \frac{|TPA(c)|}{|TPA(c)| + |FNA(c)| + |FPA(c)|}

HOTA as the average over the α grid §5.2:

HOTA \approx \frac{1}{19} \sum_{\alpha \in \{0.05, 0.10, \ldots, 0.95\}} HOTA_\alpha

MOTSA, the mask version of MOTA §5.3.1:

MOTSA = 1 - \frac{|FN| + |FP| + |IDS|}{|M|}

AMOTA, as printed in the survey §5.3.2:

AMOTA = \frac{1}{L} \sum_{r} \left(1 + \frac{FP_r + FN_r + IDS_r}{num\_gt}\right)

Motion-centric errors for autonomous driving, with v_pred and v_gt the predicted and true velocity vectors §5.4.2:

VAE = \arccos\left(\frac{v_{pred} \cdot v_{gt}}{\|v_{pred}\| \, \|v_{gt}\|}\right)

VNE = \|v_{pred}\| - \|v_{gt}\|

VIR = \frac{\text{Count}(v_{pred} \cdot v_{gt} < 0)}{\text{Total velocity predictions}}

Track-mAP takes precision and recall at each rank n, and interpolates precision as the maximum over all later ranks §5.1.4. The text gives no closed-form equation for the final score.

Hyperparameters

The survey trains nothing, so training values do not apply. The metric parameters it does state are below.

NameValueWhereNotes
HOTA localisation thresholds α0.05 to 0.95 in steps of 0.05 (19 values), averaged uniformly§5.2the integral is approximated by this sum
AMOTA recall points Lnot stated§5.3.2L is defined only as "the total number of recall values"
Track-mAP similarity threshold α_trnot stated§5.1.4threshold for a predicted trajectory to match a ground-truth trajectory
MOTP / MOTA / IDF1 matching thresholdnot stated§5.1no matching threshold is given
Velocity definition for motion metricsnot stated§5.4.2the source of velocity used for v_pred and v_gt is not described

Results

No experiments. The survey reports no benchmark numbers of its own. Its quantitative content is the dataset table and the metric definitions above. The tables for each approach group tab. 1–tab. 10 contain numbers reported by other papers; this card does not reproduce them.

Public datasets named in the survey, with frame counts exactly as the survey lists them tab. 11. The survey's footnotes are marked with an asterisk, dagger or double dagger:

DatasetYearFrame countNotesSource
MOT16201611,235tab. 11
MOT17201633,705tab. 11
MOT20202013,410dense pedestrian scenes §4.1tab. 11
DanceTrack2022105,000footnote: "Indoor scenes only"tab. 11 §4.2
SportsMOT2023150,000basketball, volleyball and footballtab. 11 §4.3
TeamTrack2024279,900soccer, basketball and handballtab. 11 §4.3
nuScenes20201,400,000camera, LiDAR and radartab. 11 §4.4

The survey lists 25 datasets in total tab. 11. Its other groups are KITTI, Waymo and MCTrack (a unified evaluation across driving datasets) §4.4, and emerging UAV, head, crowd and low-light datasets §4.5.

The survey also summarises applications by domain, target, sensor, priority and constraint tab. 13 and lists eleven future directions §7.

Paper vs code

No official code. The LaTeX source, the abstract and the arXiv metadata contain no repository link. Two GitHub repository searches (the title words, and a shortened title with "msu") returned no results.

Pitfalls

  • DetA and AssA disagree with their own formulas. The text calls DetA the harmonic mean of detection recall and precision §5.2.2. The formula given is |TP| / (|TP| + |FN| + |FP|) §5.2.2. The text calls AssA the harmonic mean of association recall and precision §5.2.3. The formula given is the mean of A(c) over TP matches §5.2.3. Take the definitions from the original HOTA metric, not from this text.
  • AMOTA's sign looks wrong. As printed, the error sum enters with a plus sign §5.3.2, so more errors would raise the score. The MOTA definition the survey builds on subtracts errors §5.1.2. The text does not define L or the recall grid in detail, so treat the printed formula as unverified and check it against the AMOTA source.
  • MOTP's variable is ambiguous. The text calls d an "overlap error" §5.1.1. The table calls MOTP a "localization error (e.g., IoU)" tab. 12. The text does not fix whether d is a distance or an overlap, so the sign convention depends on which one you pick.
  • LocA uses an undefined term. LocA is described as "average IoU" §5.2.1, and its integral uses S(c) §5.2.1. S(c) is not defined in the text.
  • VIR is called binary but defined as a ratio. The text calls it a "binary metric" §5.4.2. Its formula is a fraction of velocity predictions §5.4.2.
  • Saturation claims rest on one citation. The survey says MOT17 and MOT20 are near saturation and cites another work for the claim §4.1. It gives no MOTA or IDF1 numbers to show it.
  • Survey method is not stated. "Over a hundred papers" §1 is given with no inclusion rule, no search source and no date range. Treat the coverage as the authors' selection.
  • Dataset year column has mismatches. Several rows list a year that differs from the year implied by the cited work's reference key, for example a 2024 reference listed as 2020 tab. 11. Check years against the original dataset papers before citing them.
  • The HOTA formula is averaged over 19 thresholds. The survey's sum uses 19 α values §5.2. Reporting one threshold, or a different grid, gives a different number from the published HOTA.
  • Figure data is missing. The approach share chart fig. 3 has no numeric values in the text.
  • No quantitative comparison of methods. The survey does not rank trackers. Any ranking must come from the original papers, under their own protocols §4.1.

Lineage

  • Builds on: not applicable. The survey is a review and does not extend a single method.
  • Compared against: prior reviews named in the introduction §1. These cover deep-learning pipelines and benchmarks, data association, and newer end-to-end and transformer paradigms. None of them has a card in this wiki.
  • Superseded by: not stated.

Source and license

Adapted from “In Pursuit of Many: A Review of Modern Multiple Object Tracking Systems” by Mk Bashar, Samia Islam, Kashifa Kawaakib Hussain, Md. Bakhtiar Hasan, A. B. M. Ashikur Rahman, Md. Hasanul Kabir (arXiv:2209.04796), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.