StrongSORT: Make DeepSORT Great Again

In plain words

Following several objects through a video, such as pedestrians or cars, means giving each one the same ID from frame to frame. A common approach is tracking by detection: find the objects in every frame, then link the detections over time. StrongSORT starts from an older, simple tracker, DeepSORT, and replaces or improves its parts: a stronger detector, a stronger appearance model, a smoother way of remembering what each object looked like, a correction for the camera moving, and a simpler final matching step. The authors argue that this upgraded tracker is a fair baseline against which other trackers can be compared.

Two add-on steps target the usual failures. The first joins broken pieces of a track that belong to the same object. It uses only positions and timing, not appearance, so it is lightweight. The second fills gaps where the detector missed an object, by fitting a smooth curve through the known positions instead of a straight line. Both can be attached to other trackers. Combined with StrongSORT, they give the StrongSORT++ tracker.

TL;DR

StrongSORT is a tracking-by-detection tracker built on DeepSORT with a YOLOX-X detector, BoT appearance features, an EMA feature bank, ECC camera-motion compensation, NSA noise-adaptive Kalman updates, a combined appearance-and-motion cost and one global linear assignment §III-B. AFLink (an appearance-free tracklet linker) and GSI (Gaussian-smoothed interpolation) are optional post-processing steps §IV. Author-reported: StrongSORT++ reaches 64.4 HOTA and 79.5 IDF1 on the MOT17 test set under the private-detection protocol tab. IV. Read this if you need a strong tracking-by-detection baseline, or lightweight post-processing for an existing tracker. The released code differs from the paper in several details (NMS, the NSA scaling, the AFLink loss, GSI's length scale); see "Paper vs code".

Problem

Multi-object tracking has two "missing" problems §I:

• Missing association: one object is split across several tracklets. Online trackers have no global view when they associate, so this is common §I.
• Missing detection: the detector treats an object as background, usually because of occlusion or low resolution §I.

Global link models fix missing association by linking tracklets offline, but many of them need appearance embeddings and are computationally heavy §II-B. Linear interpolation fills missing detections cheaply, but ignores motion, so the filled boxes are less accurate §II-C. The authors also argue that DeepSORT underperforms mainly because its components are outdated, not because its tracking-by-detection paradigm is wrong §I.

Assumptions: experiments use the private-detection protocol on MOT17 and MOT20 §V-A. The paper reports one fixed set of hyperparameters across sequences, without per-sequence tuning §V-C.

Method

The paper has two parts. StrongSORT upgrades DeepSORT §III. StrongSORT++ adds two post-processing steps, AFLink and GSI §IV.

StrongSORT: DeepSORT review §III-A

DeepSORT has two branches §III-A fig. 2:

• Appearance: a small CNN computes a feature for each detection. Each track keeps a bank of features from its last 100 frames. The matching cost is the smallest cosine distance between a detection and the bank eq. 1.
• Motion: a Kalman filter predicts each track's box and then corrects it with the matched detection eq. 2 eq. 3 eq. 4 eq. 5 eq. 6. Squared Mahalanobis distance gates unlikely pairs.

Association runs a matching cascade that gives priority to tracks seen more recently, and solves each stage with the Hungarian algorithm §III-A.

d(i, j) = \min \{\, 1 - f_j^{\mathsf T} f_k^{(i)} \mid f_k^{(i)} \in B_i \,\} eq. 1

\hat{x}'_k = F_k \hat{x}_{k-1} eq. 2

P'_k = F_k P_{k-1} F_k^{\mathsf T} + Q_k eq. 3

K = P'_k H_k^{\mathsf T} \left( H_k P'_k H_k^{\mathsf T} + R_k \right)^{-1} eq. 4

x_k = \hat{x}'_k + K \left( z_k - H_k \hat{x}'_k \right) eq. 5

P_k = (I - K H_k) P'_k eq. 6

StrongSORT: the upgrades §III-B

The paper claims no new algorithm in this section §III-B. The changes are:

1. Detector. YOLOX-X replaces the Faster R-CNN of DeepSORT, following ByteTrack's training schedule §III-B §V-A.
2. Appearance model. BoT replaces the simple CNN §III-B. The released feature script uses FastReID's bagtricks_S50 configuration with DukeMTMC weights, and resizes each crop to 256×128 generate_detections.py generate_detections.py generate_detections.py.
3. EMA feature update. Instead of the feature bank, each track keeps one exponentially smoothed appearance vector. The smoothing factor α is 0.9 §III-B eq. 7. The feature bank is kept only when EMA is off.
4. Camera-motion compensation (ECC). The enhanced correlation coefficient model estimates a global rotation and translation between adjacent frames, using the MOTION EUCLIDEAN warp §III-B eq. 8.
5. NSA Kalman. Each measurement's noise is scaled by its detection confidence, so low-confidence detections have less influence on the update §III-B eq. 9.
6. Motion cost. The assignment cost mixes appearance and motion with weight λ = 0.98 on appearance §III-B eq. 10.
7. Vanilla matching. The matching cascade is replaced by one global linear assignment over all tracks §III-B tab. I.

e_i^t = \alpha\, e_i^{t-1} + (1 - \alpha)\, f_i^t eq. 7

E_{\mathrm{ECC}}(\mathbf p) = \left\lVert \frac{\bar{\mathbf i}_r}{\lVert \bar{\mathbf i}_r \rVert} - \frac{\bar{\mathbf i}_w(\mathbf p)}{\lVert \bar{\mathbf i}_w(\mathbf p) \rVert} \right\rVert^2 eq. 8

\tilde{R}_k = (1 - c_k)\, R_k eq. 9

C = \lambda A_a + (1 - \lambda) A_m eq. 10

The paper does not define the appearance term A_a or the motion term A_m in eq. 10 beyond "appearance cost" and "motion cost" §III-B. The released code uses the cosine distance for A_a and the squared Mahalanobis distance for A_m nn_matching.py linear_assignment.py (see Pitfalls).

Per-frame procedure, as the code runs it (ECC warp, then prediction, then matching). The paper does not state this order deep_sort_app.py:

# StrongSORT, one frame  [§III-B] [code: deep_sort_app.py#L170-L206]
# Input:  detections (box, score, feature), tracks, ECC warp for the frame
# Output: updated tracks

def strongsort_step(detections, tracks, warp):
    # 1. Keep scores >= 0.6; no tracker NMS  [§V-A] [code: opts.py#L127-L128]
    detections = [d for d in detections if d.score >= 0.6]
    # 2. Warp each track's box by the camera motion  [eq. 8]
    for track in tracks:
        track.warp_box(warp)
    # 3. Predict every track one step  [eq. 2] [eq. 3]
    for track in tracks:
        track.predict()
    # 4. Gate by Mahalanobis distance, mix costs  [eq. 10]
    cost = mixed_cost(tracks, detections, lam=0.98)
    # 5. One global assignment, max cost 0.45  [§III-B] [tab. I]
    matches = assign(cost, max_distance=0.45)
    # 6. Second pass on leftovers, IoU gate 0.7  [code: tracker.py#L119-L128]
    matches += iou_pass(tracks, detections, max_iou_distance=0.7)
    # 7. Kalman update, noise scaled by (1 - score)  [eq. 4] [eq. 9]
    for track, det in matches:
        track.kalman_update(det, noise_scale=1 - det.score)
    # 8. Smooth the appearance vector, alpha 0.9  [eq. 7]
    for track, det in matches:
        track.ema_update(det.feature, alpha=0.9)
    return tracks

StrongSORT++: AFLink §IV-A

AFLink is an appearance-free link model. It decides whether two tracklets belong to the same identity using only frame numbers and positions §IV-A fig. 3.

Input: each tracklet contributes its last 30 frames, with rows of (frame, x, y) (the top-left corner) §IV-A AppFreeLink.py. A tracklet shorter than 30 frames is zero-padded §IV-A. Architecture §IV-A §V-A:

• A temporal module per tracklet: four Conv-BN-ReLU layers with 7×1 kernels and 32, 64, 128 and 256 channels §V-A model.py model.py.
• A fusion module per tracklet: one 1×3 convolution with BN and ReLU §V-A model.py.
• The two branches do not share weights §IV-A model.py.
• Pool and squeeze each branch, concatenate, and pass through an MLP with two fully connected layers and a ReLU §IV-A §V-A model.py.

Training treats association as binary classification. The paper writes the loss as binary cross-entropy in which the two logits are x_n and 1 - x_n §IV-A eq. 11. The code uses a two-class cross-entropy instead (see Paper vs code).

L^{\mathrm{BCE}}_n = -\Big( y_n \log\tfrac{e^{x_n}}{e^{x_n} + e^{1 - x_n}} + (1 - y_n) \log\Big(1 - \tfrac{e^{1 - x_n}}{e^{x_n} + e^{1 - x_n}}\Big) \Big) eq. 11

Inference §IV-A §V-A:

1. Keep a pair only if the second tracklet starts 0 to 29 frames after the first one ends, and the L2 distance between the first tracklet's last position and the second tracklet's first position is at most 75 pixels §V-A AppFreeLink.py.
2. Score each kept pair with the network. Cost is one minus the association score AppFreeLink.py.
3. Keep pairs whose score is at least 0.95 (cost at most 0.05) §V-A strong_sort.py.
4. Solve a one-to-one linear assignment over the costs §IV-A AppFreeLink.py.
5. Give each linked later tracklet the earlier tracklet's identity, then keep one row per (frame, identity) AppFreeLink.py.

# AFLink linking of tracklets  [§IV-A] [code: AFLink/AppFreeLink.py#L72-L108]
# Input:  tracklets as rows (frame, id, x, y, w, h)
# Output: tracklets with merged identities

def link_tracklets(tracklets):
    # 1. Keep pairs with 0 <= gap < 30 frames and <= 75 px  [§V-A]
    candidates = [
        (i, j) for i, j in ordered_pairs(tracklets)
        if 0 <= start_frame(j) - end_frame(i) < 30
        and distance(end_xy(i), start_xy(j)) <= 75
    ]
    # 2. Score each pair on its last / first 30 frames  [eq. 11]
    scores = {
        (i, j): score_pair(last_30(i), first_30(j)) for i, j in candidates
    }
    # 3. Cost = 1 - score; keep cost <= 0.05  [§V-A]
    costs = {p: 1 - s for p, s in scores.items() if 1 - s <= 0.05}
    # 4. One-to-one linear assignment on the costs  [§IV-A]
    links = linear_assignment(costs)
    # 5. Later tracklet takes the earlier id  [code: AppFreeLink.py#L97-L107]
    tracklets = relabel(tracklets, links)
    # 6. Keep one row per (frame, identity)  [code: AppFreeLink.py#L66-L69]
    return deduplicate(tracklets)

StrongSORT++: GSI §IV-B

GSI replaces straight-line interpolation with Gaussian-process regression on each coordinate §IV-B. Steps §IV-B GSI.py:

1. Linearly fill gaps inside each track, but only gaps shorter than the maximum gap of 20 frames §V-A GSI.py strong_sort.py.
2. For each track, set the kernel length scale from the track length l, as λ = τ · log(τ³ / l) with τ = 10 §IV-B eq. 15 strong_sort.py.
3. Smooth each of x, y, w and h with a Gaussian process whose kernel is a radial basis function with that fixed length scale §IV-B eq. 12 eq. 13 GSI.py.
4. The posterior mean at the frames of the track gives the smoothed values §IV-B eq. 14.

The model for each trajectory §IV-B:

p_t = f^{(i)}(t) + \epsilon, \qquad \epsilon \sim \mathcal N(0, \sigma^2) eq. 12

f^{(i)} \in \mathrm{GP}\big(0,\; k(\cdot,\cdot)\big), \qquad k(x, x') = \exp\!\Big(-\frac{\lVert x - x' \rVert^2}{2\lambda^2}\Big) \quad \text{(kernel stated in the text)} eq. 13

P^* = K(F^*, F)\,\big(K(F, F) + \sigma^2 I\big)^{-1} P eq. 14

\lambda = \tau \cdot \log\!\big(\tau^3 / l\big) eq. 15

The paper does not give a value for σ §IV-B.

StrongSORT++ = StrongSORT + AFLink + GSI §IV §V-C

The released pipeline runs tracking for each sequence, then AFLink, then GSI on the output file strong_sort.py. Each step is optional in the code (--AFLink, --GSI) opts.py.

Hyperparameters

Values marked "not stated" are missing from the paper. Code values are given only as pointers for implementers. The released repository covers MOT17 and MOT20 only opts.py.

Results

Author-reported. Setup: MOT17 (seven training and seven test sequences, private-detection protocol), MOT20 (four test sequences), DanceTrack and KITTI §V-A. All experiments ran on a single V100 §V-A. The authors caution that FPS figures depend on the hardware and on whether detection time is counted §V-C. The ablations use the first half of each MOT17 training sequence for training and the last half for validation §V-A.

Ablation on MOT17 validation, StrongSORT components tab. I. Each row adds one component to the row above. Best row in bold in the paper.

AFLink and GSI applied to six trackers, MOT17 validation tab. II. The Δ values in the paper are relative to the first row of each block (no AFLink, no GSI).

Matching cascade versus vanilla matching, MOT17 validation tab. III. Δ is relative to the cascade row.

MOT17 test, private detections tab. IV. Online rows first, then offline rows. Values as printed in the paper. A dash means the paper gives no value.

MOT20 test, private detections tab. V.

DanceTrack test tab. VI. Only the bold-marked rows are the paper's own method.

KITTI test, Car and Pedestrian tab. VII. HOTA, MOTA, AssA and IDs for each class.

The paper's summary: StrongSORT++ ranks first on MOT17 for HOTA, IDF1, AssA and DetA, and second for MOTA and IDs §V-C. On MOT20 it ranks first for HOTA, IDF1 and AssA with far fewer IDs, but its MOTA and DetA are lower; the paper attributes this to missed detections from the high score threshold, with 117,920 false negatives against ByteTrack's 87,594 §V-C. On KITTI it is comparable for cars and better for pedestrians than PermaTrack §V-C.

Ablation findings in the text §V-B: BoT gives +2.2 IDF1, ECC +0.2 IDF1 and +0.3 MOTA, NSA +0.4 HOTA, EMA +0.4 IDF1 with +1.2 FPS, the motion cost +0.8 IDF1, and vanilla matching +1.4 IDF1 for the strongest tracker. The authors conclude that the matching cascade's prior helps weak trackers but limits strong ones §V-B.

Paper vs code

The official repository is dyhBUPT/StrongSORT at commit ee995076da5083e28d0da1f885297df62705ebd7, under GPL-3.0 LICENSE. The owner is the first author's account, and the abstract links it. The abstract also links open-mmlab/mmtracking, a third-party framework, which this card does not use. The license is copyleft, so the card describes the code in its own words and cites paths and lines; nothing from the code is quoted.

• Tracker-level NMS is switched off. The paper sets NMS to 0.8 §V-A. The released tracker sets the threshold to 1.0 opts.py. The NMS step suppresses only when overlap is above the threshold preprocessing.py, and overlap is at most 1, so nothing is suppressed. The 0.8 value is used when exporting detections, with confidence 0.6 AuxiliaryTutorial.md.
• The matching threshold depends on the flags. The paper gives 0.45 §V-A. The code uses 0.4 for BoT and adds 0.05 when motion cost is on, so 0.45 holds only with the full StrongSORT configuration opts.py.
• NSA scales the standard deviation, so the variance scales by (1 − c)². The paper writes R̃ = (1 − c)R §III-B eq. 9. The code multiplies each standard deviation term by (1 − c) kalman_filter.py, which squares the factor in the covariance. The rule runs only in the measurement update kalman_filter.py. The gating distance calls the same projection without a confidence, so gating uses the unscaled noise kalman_filter.py.
• EMA re-normalizes the feature. Eq. 7 has no normalization §III-B. The code scales the smoothed vector to unit length after each update track.py.
• The motion term is an unnormalized squared Mahalanobis distance. The paper does not define A_m §III-B eq. 10. The code uses the squared Mahalanobis distance from the Kalman projection kalman_filter.py, gates entries above 9.4877 to a large cost linear_assignment.py linear_assignment.py, and mixes the costs after gating linear_assignment.py.
• Input scaling for the appearance model is not stated. The released feature script resizes crops, converts them to tensors and multiplies by 255, with the mean and standard deviation normalization commented out generate_detections.py generate_detections.py. The paper does not state the input convention.
• Camera compensation runs before the prediction, and ECC parameters are missing from the paper. The loop warps track boxes first, then predicts deep_sort_app.py. The paper gives only the warp mode. The code uses eps 1e-5, 100 iterations and a 0.1 downscale ecc.py ecc.py, and it uses the identity warp when the matrix is far from identity track.py.
• The AFLink loss is a two-class cross-entropy, not the paper's formula. Eq. 11 writes the logits as x and 1 − x §IV-A. The code uses a two-output classifier with standard cross-entropy train.py train.py, and a softmax at inference model.py.
• AFLink training data is not named in the paper. The code trains on the seven MOT17 training sequences that use the FRCNN detections, and only on tracks of at least 30 frames dataset.py dataset.py. The test sequences are excluded dataset.py. The file names say "half" dataset.py, which suggests the same split as the ablation in §V-A, but the repository does not ship the split files. The code's AFLink training data appears to come from the same sequences the ablation validates on, so ablation-level AFLink gains may be optimistic. The MOT17 test results do not share this issue.
• Reversed pairs are negatives. The code labels each pair in reversed order as negative, besides the two perturbed negatives and the positive pair dataset.py. The 1:3 positive-to-negative ratio matches the paper §V-A.
• Padding happens before normalization. The paper says short tracklets are zero-padded §IV-A. The code inserts zeros before the joint min-max normalization of the pair dataset.py dataset.py, so padded entries are not zero after normalization.
• The accept threshold is off by a boundary. The paper says a score larger than 0.95 §V-A. The code keeps cost ≤ 0.05 for the matrix AppFreeLink.py and requires cost < 0.05 for the final assignment AppFreeLink.py. Effects are negligible, but the boundary differs.
• The GSI window is one frame shorter. The paper says the maximum gap is 20 frames §V-A. The code fills a gap when the frame difference is between 2 and 19, so at most 18 frames are inserted GSI.py.
• GSI clips the length scale. Eq. 15 has no bounds §IV-B. The code clips it to [1/τ, τ²] = [0.1, 100] GSI.py. Eq. 15 is at most 0.1 for tracks longer than about 990 frames, and the unclipped formula is zero at 1000 frames and negative after that. The clip keeps the kernel valid.
• GSI's noise term is the scikit-learn default. Eq. 14 uses σ²I with no value §IV-B. The code creates a Gaussian-process regressor with a fixed RBF kernel and no explicit σ, so scikit-learn's default alpha of 1e-10 applies GSI.py.
• The DanceTrack and KITTI settings are not in the repository. The released configuration covers MOT17 and MOT20 only opts.py. The DanceTrack and KITTI settings §V-C have no code here.
• The detector and the appearance model are not in the repository. The repository reads detections and features from files README.md. The YOLOX training schedule §V-A and the BoT training §V-A are not reproduced by this code.

Pitfalls

• The motion term needs a scale check. A_m is the raw squared Mahalanobis distance, and the gate threshold is 9.4877 linear_assignment.py. With λ = 0.98, the motion term contributes at most about 0.19 inside the gate, which is small next to the appearance term (cosine distance, 0 to 2). A reimplementation that normalizes the motion term differently will change which matches survive the 0.45 threshold.
• Gating and update disagree on the noise. NSA applies only in the update kalman_filter.py and is absent from gating kalman_filter.py. A reimplementation that applies NSA in both places will not match the released behaviour.
• Feature input scale matters for the pretrained weights. If you reuse the released features or re-extract them, match the 0–255 input range with no mean or standard deviation normalization generate_detections.py generate_detections.py. The paper says nothing about this.
• The 1.0 tracker NMS is not a bug to fix. Turning NMS on at 0.8 inside the tracker changes the released results. The 0.8 value belongs to the detector export AuxiliaryTutorial.md.
• AFLink needs a GPU in the released code. The class moves the model and inputs to CUDA AppFreeLink.py AppFreeLink.py. A CPU port needs changes.
• AFLink's windows are tied to MOT17's frame rate and resolution. The 30-frame and 75-pixel gates §V-A are pixel and frame values, and the paper reports different AFLink thresholds for DanceTrack §V-C. Recalibrate them for other data.
• Deduplication keeps the first row. After relabelling, the code keeps the first row for each (frame, identity) pair in file order AppFreeLink.py. Duplicates are dropped without a score check.
• GSI rewrites every box. The smoothing runs over all rows of each identity, not only the gaps GSI.py. Observed boxes change too. The paper describes the same behaviour §IV-B, and the velocity figure is the authors' evidence for it fig. 5.
• The ablation's AFLink validation may be optimistic. See the AFLink training-data bullet in Paper vs code. Use the MOT17 test numbers for comparisons.
• The ablation's conclusion depends on the detector. The vanilla matching gain is +1.4 IDF1 for StrongSORTv5 but −1.1 for DeepSORT tab. III. The authors attribute the reversal to a stronger tracker §V-B.
• FPS is not comparable across rows. The authors say FPS depends on the device and on whether detection time is counted §V-C. The table mixes numbers from different papers.
• Numbers are author-reported under the private-detection protocol. Compare only with rows under the same protocol.
• MOTA trails the detection-heavy trackers. On MOT20, StrongSORT++ has far fewer IDs but more false negatives than ByteTrack §V-C. Choose StrongSORT++ for association-heavy metrics, not for detection recall.

Lineage

• Builds on DeepSORT §I §III-A, which has no card here. Its matching cascade is replaced by a global assignment §III-B.
• Uses YOLOX-X §III-B, whose training schedule follows ByteTrack §V-A. Neither has a card here.
• Uses BoT features §III-B (FastReID) generate_detections.py, EMA from Wang et al. §III-B, NSA from GIAOTracker §III-B and ECC camera compensation §III-B. None has a card here.
• Compared with global link models TNT, TPM, ReMOT, GIAOTracker and LGMTracker §II-B. None has a card here.
• Compared with interpolation methods V-IOUTracker, MAT and MAATrack §II-C. The Gaussian-process interpolation of Zhu et al. is the closest prior work §II-C.
• Compared against in tables: CenterTrack, TransTrack, FairMOT in tab. II; SORT, Tracktor++, ByteTrack, OC-SORT, MAATrack and others in tab. IV and tab. V; PermaTrack and QDTrack in tab. VII.
• Superseded: not known from the content already in this wiki.