SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes

In plain words

Tracking many people through a video means finding each person in every frame and keeping the same ID on them over time. Most public benchmarks track pedestrians on streets or dancers, who move slowly or in predictable ways. Players on a basketball, volleyball or football court are different: they sprint, stop and change speed quickly, and many of them wear similar clothes. Existing trackers, which rely heavily on predicting where each person will be next, struggle with this.

This card covers two things. The first is SportsMOT, a new labelled video dataset of professional games in these three sports, in which only the players on the court are annotated. The second is MixSort, a tracking method that adds an appearance check to existing trackers. The check uses a small network that compares a stored picture of each tracked player with the candidate detections in the new frame, and its score is blended with the usual overlap-of-boxes score. On the dataset, the authors report that this helps the trackers keep identities more reliably.

TL;DR

SportsMOT (240 sequences, 3 sports, 45/45/150 train/val/test split) is a dataset for tracking only on-court players, where motion is fast and variable and appearance is similar but distinguishable §1 §3. MixSort is a plug-in for tracking-by-detection trackers (ByteTrack, OC-SORT) that fuses an IoU score with a MixFormer heatmap similarity §4 eq. 1. The author-reported test-set gains are modest: MixSort-OC 74.1 HOTA vs 73.7 for OC-SORT and MixSort-Byte 65.7 vs 64.1 for ByteTrack, both with train+val training tab. 3. Read the Pitfalls section before using the numbers.

Problem

• Benchmarks such as MOT17, MOT20 and DanceTrack track pedestrians or dancers, and the motion models of popular trackers assume slow, near-linear motion §1 §3.2.
• In sports the authors observe fast, variable-speed motion, which makes motion-based association hard: a Kalman-filter IoU is lower on SportsMOT than on MOT17 and DanceTrack fig. 2 fig. 3.
• Players in one team wear similar outfits, so appearance is less discriminative than in MOT17 but more discriminative than in DanceTrack §1 fig. 4.
• The authors state that the main difficulty on SportsMOT is association (IDF1 and AssA are lower than on MOT17 while DetA is high) rather than detection §1.

Method

The paper has two parts. The dataset part (§3) describes video selection, the annotation rules and statistics. The method part (§4) describes MixSort.

Dataset construction §3.1

• Source: overhead broadcast shots of professional games (NCAA, Premier League and Olympic footage) taken from the MultiSports dataset. Only overhead shots are used; each clip is checked for abrupt viewpoint switches §3.1.
• 240 sequences, each 720P at 25 FPS, across football, basketball and volleyball §3.1.
• Annotators box the whole visible body of each athlete (limbs and torso, not the ball), predict boxes for occluded athletes when any body part is visible, and skip athletes with half the torso out of view. Each player keeps one ID per clip §3.1.
• A labelling tool propagates boxes with the single-object tracker KCF; annotators then refine boxes and IDs. Boxes with width or height below 5 pixels are deleted §3.1.

Dataset split and statistics §3.2 tab. 1 tab. 2

• Split: 45 train, 45 validation and 150 test sequences §3.2.
• Totals: 150,379 frames, 1,629,490 boxes, 3,401 tracks tab. 1.
• Per category tab. 2: basketball, volleyball and football average 845.4, 360.4 and 673.9 frames per sequence respectively (the table gives per-category averages, not totals).

Evaluation metrics §3.3

The authors recommend HOTA, AssA and IDF1 as the main metrics, because MOTA weights detection accuracy more than association §3.3.

MixFormer for appearance-based association §4.1

• MixSort uses a MixFormer-style network (from the authors' earlier work, cui2022mixformer). Given a template (the tracked player's crop) and a search region (a crop of the current frame), it produces a heatmap whose peak marks the template's centre in the search region §4.1 fig. 5.
• The original MixFormer predicts template corners. The authors replace the corner head with a heatmap head that predicts the template centre and gives a confidence heatmap §4.1.
• To speed up inference and suit MOT, the number of mixed attention modules is reduced from 12 to 4 §4.1.

Association strategy §4.1 eq. 1

For each existing track t:

1. Predict its new location with a Kalman filter §4.1.
2. Crop the current frame centred on that prediction at a set scale; this is the search region s §4.1.
3. Run MixFormer on (template of t, s) to get heatmap H §4.1.
4. For each detection d whose centre lies in s, set the visual similarity V[t, d] to the heatmap value at d's centre. Other detections get 0 §4.1 fig. 5.
5. Fuse with IoU and solve the assignment with the Hungarian algorithm §4.1 §4.2.

M = \alpha \cdot \mathrm{IoU} + (1 - \alpha) \cdot V eq. 1

where \alpha is a weight coefficient and V is the visual-similarity matrix from MixFormer §4.1.

MixSort tracking loop §4.2 fig. 6

The method is described as generic over tracking-by-detection trackers (ByteTrack and OC-SORT are named) §4.2. Per frame:

# Associate detections with tracks in one frame  [§4.2]
# Input:  tracks, detections (boxes, scores), frame
# Output: matched (track, detection) pairs

def mixsort_step(tracks, detections, frame):
    # 1. Predict each track's new location with a motion model  [§4.2]
    predicted = [kalman_predict(track) for track in tracks]

    # 2. Crop a search region around each prediction and run MixFormer
    #    with the track's template  [§4.1]
    heatmaps = [mixformer(track.template, crop(frame, box))
                for track, box in zip(tracks, predicted)]

    # 3. Visual similarity: heatmap value at each detection centre that
    #    lies inside the search region, 0 otherwise  [§4.1] [fig. 5]
    visual = visual_similarity(heatmaps, detections)

    # 4. Fuse the visual similarity with the IoU matrix  [§4.1] [eq. 1]
    mixed = alpha * iou(predicted, detections) + (1 - alpha) * visual

    # 5. Associate tracks and detections with the Hungarian algorithm  [§4.2]
    pairs = hungarian_match(mixed)

    # 6. Matched tracks: update motion state and, under a condition, the
    #    template  [§4.2] [§4.3]
    update_matched(pairs)

    # 7. Unmatched tracks are kept until a limit is reached; unmatched
    #    detections above a score threshold start new tracks  [§4.2]
    return pairs

Notes on the pseudocode: the paper has no algorithm box, so it is a condensed rendering of the §4.2 text. The alpha in step 4 is the fusion weight from eq. 1. The number of frames for the "limit" in step 7 and the score threshold are not stated in the paper.

Template update §4.3

For each track there is one template. When a detection is matched to a track, the template is replaced by the new detection, but only if the ratio of the detection's uncovered area (area not overlapping any other detection) is larger than a threshold. The threshold value is not stated §4.3.

MixFormer training §4.3 eq. 2 eq. 3

• Ground-truth trajectories of every player are converted to the TrackingNet format §4.3.
• Each box centre (c_x, c_y) is mapped to the low-resolution heatmap and expanded with a 2D Gaussian §4.3 eq. 2:

h_{xy} = \exp\left(-\frac{(x - c_x)^2 + (y - c_y)^2}{2\sigma^2}\right) eq. 2

The paper says \sigma is adaptive to the size of the bounding box §4.3.

• Loss is pixel-wise logistic regression with focal loss, CornerNet-style §4.3 eq. 3:

L = -\sum_{xy} \begin{cases} (1 - \hat h_{xy})^{\gamma} \log(\hat h_{xy}) & h_{xy} = 1 \\ (1 - h_{xy})^{\beta} (\hat h_{xy})^{\gamma} \log(1 - \hat h_{xy}) & \text{otherwise} \end{cases} eq. 3

With \gamma = 2 and \beta = 4 following CornerNet §4.3.

Detector §5.1

YOLOX is the detector, as in ByteTrack and OC-SORT. The COCO-pretrained model is trained on CrowdHuman for 80 epochs and then on SportsMOT for 80 epochs; the other settings follow ByteTrack §5.1.

Hyperparameters

Values as stated in the paper unless marked "code". Where the paper and the official code disagree, both values are listed.

Results

Author-reported. The table below is the main benchmark on the SportsMOT test set tab. 3. Rows marked "Train" are trained on SportsMOT train only; "Train+Val" rows use the train and validation splits. The paper's "(+x)" marks are the gains over the matching baseline.

Source: tab. 3. Bold in the original marks the best per column; the table above omits bold.

MOT17 (private detector) tab. 5

Source: tab. 5. The paper's text says MixSort outperforms these trackers in HOTA, IDF1 and AssA on MOT17 §5.2.

Fusion weight alpha per sport (HOTA, SportsMOT test, OC-SORT + MixSort) tab. 4

Ablation on the SportsMOT test set (trained on train+val) tab. 6

Source: tab. 6. The paper's text uses the first row (IoU only, 71.5) and the third row (IoU+Kalman, 64.1) to show that the Kalman filter hurt on SportsMOT §5.3.

Per-category and SoccerNet comparison (SportsMOT test, MixSort-Byte) tab. 7

Source: tab. 7. Models trained on SportsMOT train+val and on SoccerNet train respectively.

DanceTrack validation tab. 8

Source: tab. 8. Trained on DanceTrack train. The paper states alpha 0.9 gave the highest HOTA among {0.6, 0.7, 0.8, 0.9, 0.95} tab. 8.

ReID model comparison (SportsMOT test) tab. 9

Source: tab. 9. The paper states MixSort-Byte improves ByteTrack+ReID by 0.9, 1.9 and 1.4 on HOTA, IDF1 and AssA §5.3.

Dataset statistics tab. 1 tab. 2

Source: tab. 1.

Source: tab. 2.

Paper vs code

Official code: the SportsMOT project page links the SportsMOT repository and calls the MixSort repository the proposed baseline tracker. The MixSort repository at https://github.com/MCG-NJU/MixSort is used here (MIT license, as recorded in code_license, commit a078f5bf6ae9fbeecbc1384479d5f02ab8b9e7f6). Its README cites this paper. The repository's LICENSE file names "Copyright (c) 2021 Yifu Zhang", which indicates it is derived from ByteTrack code; the card uses only the authors' repository and the code cited below. The SportsMOT dataset repository was not cloned.

The code contains the MixFormer and tracker parts. Discrepancies found:

• Learning rate. Paper: 1e-4, dropped to 1e-5 at epoch 200 §5.1. Code: LR 0.0004 with a drop at epoch 100 and decay 0.1 baseline.yaml. The code's decay factor (0.1) matches the paper's 1e-4 to 1e-5 ratio; the start and drop epoch do not.
• Optimizer. Paper: ADAM §5.1. Code: AdamW, with weight decay 1e-4 base_functions.py baseline.yaml.
• Gaussian width. Paper: sigma adapts to the bounding-box size §4.3 eq. 2. Code: sigma = sigma_factor times the square root of the feature-map area, which does not depend on the box size processing_utils.py.
• Loss. Paper: focal loss only §4.3 eq. 3. Code: the heatmap actor's loss is the modified focal term mixformer.py (defined at mixformer.py). The config also lists IOU_WEIGHT 2.0 and L1_WEIGHT 5.0 baseline.yaml; these are not used in the heatmap actor lines read here.
• Pretraining. Paper: VOT-trained backbone §5.1. Code: MixFormerOnlineScore.pth.tar baseline.yaml. The code does not name the pretraining data.
• Fusion weight. Paper: alpha is tuned per sport §5.3 tab. 4 and the main-table value is not stated. Code: default 0.6 track_mixsort_oc.py track_mixsort.py; the MOT17 evaluator overrides it to 0.8778 mot_evaluator.py.
• Fusion formula. Paper: alpha * IoU + (1 - alpha) * V eq. 1. Code (Byte tracker): the same formula mixsort_tracker.py. Code (OC-SORT tracker): the IoU term includes an extra angle-difference term inside the alpha-weighted part association.py. The paper does not mention the angle term.
• Interpolation. Paper: maximum gap 20 §5.1. Code: the condition is strict (gap < 20) interpolation.py. The minimum track length n_min (25 by default) is not in the paper interpolation.py.
• Template update. Paper: threshold on uncovered-area ratio, value not stated §4.3. Code: a mix_iou threshold, default 0.2 track_mixsort_oc.py, compared to a per-detection value mixsort_oc_tracker.py. The code does not show that this value is the uncovered-area ratio; not verified.
• Batch size and factors. Not stated in the paper. Code: batch 96, search factor 4.0, template factor 1.5 baseline.yaml baseline.yaml baseline.yaml.
• Consistent items. Search size 224, template size 96, max sample interval 10, 300 epochs, weight decay 1e-4, focal gamma 2 and beta 4 match between paper and code baseline.yaml baseline.yaml mixformer.py.

Not checked: the detector configuration files (exps/example/mot/yolox_x_sportsmot.py), the dataset conversion scripts, and whether the 12-to-4 attention reduction exists in the code.

Pitfalls

• Inconsistent dataset ratios. The abstract says the dataset has almost 15x the frames of MOT17 and 3x its boxes. §3.2 says frames are 13.4x MOT17, which matches 150,379 / 11,235. tab. 1 gives 1,629,490 / 292,733 ≈ 5.6x boxes, not 3x. Check numbers against the table, not the prose.
• Numbers that differ across tables. MixSort-Byte appears with MOTA 96.2 and 2472 IDs in tab. 3 but 96.1 and 2469 in tab. 6 and tab. 7. The best-MixSort ablation row (73.8 HOTA) is not the same configuration as the main table (74.1 HOTA for MixSort-OC). The ablation trains on train+val and states this in its caption tab. 6.
• Alpha is not one number. The paper tunes alpha per sport tab. 4, and the code defaults to 0.6 track_mixsort_oc.py. Using the defaults does not reproduce the per-sport optima; the paper does not say which alpha produced tab. 3.
• Kalman filter. The paper says the Kalman filter hurt on SportsMOT §5.3 tab. 6. Any reimplementation should check the motion model before it adds one.
• Gains are small. On the test set, the MixSort gains over the base trackers are 0.4 to 1.6 HOTA tab. 3. Comparisons use different training data (train vs train+val) in the same table; compare within a training setup.
• Metric caveat. The paper names HOTA, AssA and IDF1 as the main metrics §3.3; several quoted gains are in AssA and IDF1 rather than HOTA.
• Post-processing. Linear interpolation with max gap 20 is applied to every tracking result §5.1. The code has a separate script for it with its own minimum-length setting interpolation.py; the paper does not say which setting produced the reported numbers.
• Sigma and sizes. The Gaussian width is described as box-adaptive, but the code uses feature-map area processing_utils.py. Test the heatmap target scale before relying on the paper's description.
• Annotation. Boxes smaller than 5 pixels in width or height are removed §3.1. Occluded players are annotated by prediction; the paper does not give the rule for when a prediction is "too uncertain".
• Detector training. Detector hyperparameters are "as ByteTrack" §5.1. A reimplementation must take them from ByteTrack's configuration; the paper does not restate them.
• Licensing. The paper is CC BY 4.0. The code is MIT, but its LICENSE names a 2021 copyright holder (derived from ByteTrack). Keep the attribution.

Lineage

• Builds on: MixFormer (cited as cui2022mixformer), with the corner head replaced by a heatmap head and the mixed-attention count reduced from 12 to 4 §4.1.
• Compared against: ByteTrack, OC-SORT, CenterTrack, FairMOT, QDTrack, TransTrack, GTR, MOTR, plus DeepSORT's ReID model on SportsMOT tab. 3 tab. 5 tab. 9.
• Motion and association cues: Kalman filter, IoU matching and the Hungarian algorithm §2 §4.1.
• Focal loss from CornerNet, with gamma 2 and beta 4 §4.3.
• Superseded by: not known from this wiki's content.