TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking

In plain words

Following people or cars through a video means deciding, frame after frame, which new detection belongs to which track already in progress. Earlier transformer-based trackers treated every track and every detection as one long list and let attention compare them all. That works poorly when a scene holds many objects, and it is costly.

TransMOT instead arranges the tracks and the new detections as graphs, where two objects are linked only if their boxes overlap, and the link is weighted by how much they overlap. Attention then runs mostly along those links, first within each frame and then across the recent frames for each track. A matching step produces a table of which detection continues which track, or starts a new one, or ends. A separate cascade stage handles the easy cases first, such as confident detections, and the cases where an object was hidden for a while.

The result is a tracker that the authors report as more accurate and cheaper to run than a plain transformer over the same objects, evaluated on the MOT15, MOT16, MOT17 and MOT20 pedestrian benchmarks.

TL;DR

• Online multi-object tracker: each frame, the current tracks are matched to new detections §3.
• Core idea: tracks and detections are sparse IoU-weighted graphs; a graph transformer encoder, a temporal transformer encoder and a graph transformer decoder produce an assignment matrix §4.
• A three-stage cascade (Kalman-based motion matching, transformer association, occlusion and duplicate handling) sits around the model §4.4.
• Author-reported state of the art on the MOT15, MOT16, MOT17 and MOT20 test sets; the paper's own comparison is against the methods in Tables 1 to 4 §5.2.
• Reimplementation risk is high: the loss has a sign question, and several cascade thresholds are not stated (see Pitfalls).

Problem

Tracking-by-detection needs a model that links the existing tracks to the detections in each frame §1. The authors argue that a general transformer over all objects of a video (a) ignores the spatial structure of the scene, (b) needs much data and computation to learn long temporal dependencies, and (c) builds on a DETR-based detector that is not state of the art for MOT §1. The model is meant for online tracking with a fixed number of past frames, so tracks that were occluded for longer than that window are not handled by the model itself [§3, §4.4].

Method

Notation. Let the tracked set at frame t-1 have N tracklets, each with states over the last T frames, and let frame t have M candidate detections §3. The model outputs an extended assignment matrix Abar of size (M+1) x (N+1). The extra row is a virtual sink (a track that exits or is occluded), and the extra column is a virtual source (a detection that starts a new track) §4.2 §4.3.

Assignment objective. The association is first written as a constrained maximisation of pairwise affinities, with one-to-one constraints on rows and columns eq. 1 eq. 2. The model replaces this with a single mapping Abar = Phi(tracklets, candidates) §3. During training, the one-to-one constraint is relaxed so that each detection goes to exactly one track or the virtual source, while a track may take several detections; each row then is a distribution over N+1 options §4.3.

Step 1. Tracklet graphs §4.1 fig. 2. For each of the last T frames, the tracklets form a graph. Nodes are tracklet states, an edge exists if the two boxes have IoU greater than zero, and the edge weight is that IoU. The node features are a learned linear embedding into D dimensions §4.1. The paper's node feature is the visual feature (SiamFC) concatenated with normalised box coordinates §5.1.

Step 2. Spatial graph transformer encoder layer §4.1.1 eq. 3. Multi-head graph attention runs independently for each frame. The attention weights of head i are the scaled dot-product scores multiplied element-wise by the graph weights, then passed through a softmax eq. 3. The values come from a graph convolution over the node features rather than a linear map §4.1.1. Heads are concatenated and projected by an output matrix W^O, then followed by a linear feed-forward layer and a normalisation layer §4.1.1. The paper uses 8 heads and one layer of ChebConv with neighbourhood distance 2 §5.1.

Step 3. Temporal transformer encoder layer §4.1.2. The output of step 2 is re-ordered from frame-major to tracklet-major, so that each tracklet gets a sequence of T frame embeddings. A standard transformer encoder then attends over time, separately for each tracklet. Its output is the encoder output F_out^en §4.1.2.

Step 4. Candidate graph and sink §4.2 fig. 3. The M candidates form a graph built the same way as in step 1 (edges where IoU is above zero, weight equal to IoU). A learnable sink embedding f_snk is added as one more node, and it is connected to every other node with fixed weight 0.5 §4.2.

Step 5. Graph decoder §4.2. The candidate node features (now M+1 nodes) pass through graph multi-head attention, the same kind as in step 2. The result is copied N+1 times so that each candidate can be compared with each tracklet and with the virtual source. A learnable embedding serves as the virtual source and is appended to the tracklet embedding, so the tracklet side has N+1 entries §4.2. Multi-head cross attention is computed between these two sides, followed by a feed-forward and normalisation layer §4.2. The paper uses one virtual source, not several as in earlier transformer trackers, because it performed about as well and was cheaper §4.2.

Step 6. Assignment output §4.2. A linear layer and a softmax turn the decoder output into Abar of size (M+1) x (N+1).

Step 7. Cascade association at inference §4.4 fig. 4. Three stages:

1. Motion matching for confident tracks. A Kalman filter predicts boxes for tracks that were matched in each of the last K_r frames. Predicted and candidate boxes are matched with the Hungarian algorithm by IoU. Pairs with IoU above tau_M are accepted. Low-confidence candidates that stay unmatched are removed §4.4.
2. TransMOT association. The model runs on the remaining tracks and candidates. The block of Abar that links real tracks to real candidates is soft, so a bipartite matching is applied to it. Only matches whose score exceeds a threshold are kept §4.4. The paper does not name the matching algorithm here.
3. Long-term occlusion and duplicates. Tracks that were occluded for more than T frames (outside the model's window) are matched by a cost that adds the Euclidean distance between stored visual features and a normalised top-distance D_top between the last visible box and the candidate §4.4 [the formula for D_top is unnumbered in the paper]. Candidates that lie inside an already matched box are removed as duplicates, using intersection over the candidate's own area as the cost §4.4. Remaining candidates start new tracks. Tracks not updated for more than K_p frames are removed; tracks not updated for fewer frames are marked "occluded" §4.4.

Training objective §4.3 eq. 4. The loss has three parts: (i) cross-entropy on the detection rows, (ii) a sink term for the sink row, and (iii) a sink term for the non-sink entries. Terms (ii) and (iii) are weighted by lambda, whose value is not stated §4.3. Training uses random windows of T+1 consecutive frames §4.3. Ground-truth boxes are swapped for detector boxes that match them by IoU, so the model sees detector noise §4.3. Detector IDs are remapped to the {0, ..., N} index space, where a detection either matches a tracklet or the virtual source §4.3.

Formula, eq. 4 as typeset eq. 4:

L = -1/M * sum_m y_m log(abar_m)
    + lambda/N * sum_n y_n^snk log(1 / (1 + exp(-a'_n)))
    + lambda/N * sum_n (1 - y_n^snk) log(exp(-a'_n) / (1 + exp(-a'_n)))

where y_m is the index of the target column for detection m, abar_m is row m of Abar, y_n^snk flags whether tracklet n is the sink target, and a'_n is the sink logit for tracklet n §4.3.

Pseudocode

The following sketches the encoder, the decoder and the loss, written from the text and the equations above. Names are ours; shapes follow the paper's tensor notation.

# Spatial-temporal encoder over tracklet graphs  [§4.1] [eq. 3]
# Input:  tracklet_feats (N, T, D0), iou (T, N, N) sparse
# Output: encoded (N, T, D), D = 1024  [§5.1]

def encode_tracklets(tracklet_feats, iou):
    # Linear embedding of each node, independently  [§4.1]
    embedded = linear_embed(tracklet_feats)           # (N, T, D)
    spatial = []
    # Spatial graph attention, one graph per frame  [§4.1.1] [eq. 3]
    for t in range(T):
        # 8 heads, concatenated and projected by W^O  [§5.1] [§4.1.1]
        scores = scaled_dot(embedded[:, t], embedded[:, t])
        # Element-wise product with IoU weights, then softmax  [eq. 3]
        attention = softmax(scores * iou[t])
        # Graph convolution gives the value features  [§4.1.1] [§5.1]
        values = graph_conv(embedded[:, t], iou[t])
        spatial.append(attention @ values)
    spatial = stack(spatial, axis=1)                  # (N, T, D)
    # Feed-forward and normalisation; sizes not stated  [§4.1.1]
    spatial = feed_forward_and_norm(spatial)
    # Temporal encoder over T for each tracklet  [§4.1.2]
    ordered = transpose(spatial, 0, 1)                # (T, N, D)
    temporal = temporal_transformer(ordered)          # over T, per track
    return transpose(temporal, 0, 1)                  # (N, T, D)

# Candidate decoder and assignment matrix  [§4.2]
# Input:  candidates (M, D0), candidate_iou (M, M), encoded (N, T, D)
# Output: assignment (M+1, N+1), each row sums to 1 (softmax over columns)

def decode_assignment(candidates, candidate_iou, encoded):
    # Embed candidates, then add the learnable sink node  [§4.2]
    nodes = concat(linear_embed(candidates), learned_sink)   # (M+1, D)
    # Sink connected to all nodes with weight 0.5  [§4.2]
    graph_weights = add_sink_edges(candidate_iou, sink_weight=0.5)
    # Graph multi-head attention over candidates  [§4.2]
    attended = graph_attention(nodes, graph_weights)         # (M+1, D)
    # Learnable virtual source appended on the tracklet side  [§4.2]
    tracklets = concat(encoded_flat, learned_source)         # (N+1, ..., D)
    # Duplicate candidates over N+1 columns, then cross attention  [§4.2]
    queries = repeat(attended, times=N + 1, axis=1)          # (M+1, N+1, D)
    cross = cross_attention(queries, tracklets)
    cross = feed_forward_and_norm(cross)
    # Linear layer and softmax over the N+1 columns  [§4.2]
    logits = linear_out(cross)                               # (M+1, N+1)
    return softmax(logits, axis=1)

# Training loss  [§4.3] [eq. 4], as typeset
# Inputs: assignment (M+1, N+1), target_cols (M,) index into N+1 columns,
#         sink_flags (N,) in {0, 1}, sink_logits (N,)
# lam: weight, value not stated  [§4.3]

def training_loss(assignment, target_cols, sink_flags, sink_logits, lam):
    # Cross-entropy over the detection rows  [§4.3] [eq. 4, term 1]
    row_probs = assignment[:M][range(M), target_cols]
    detection_term = -mean(log(row_probs))
    # Sink term for tracklets flagged as sink  [eq. 4, term 2]
    sink_pos = lam * mean(sink_flags * log(sigmoid(sink_logits)))
    # Sink term for the remaining tracklets  [eq. 4, term 3]
    sink_neg = lam * mean((1 - sink_flags) * log(1 - sigmoid(sink_logits)))
    # Signs as typeset; see Pitfalls before using  [§4.3]
    return detection_term + sink_pos + sink_neg

Hyperparameters

For the ablation, the detector was trained on CrowdHuman only §5.3, so ablation numbers are not comparable with Tables 3 and 4.

Results

Author-reported, on the MOT challenge test sets. Metrics: IDF1, MOTA, MT (mostly tracked, higher is better), ML (mostly lost, lower is better), FP, FN and IDS (identity switches, lower is better). Baseline rows are as listed in the cited tables; bold marks the best entry per column, as the source's bolding intends.

MOT15, private detection track tab. 1

MOT16, private detection track tab. 2

MOT17, public and private detection tracks tab. 3

MOT20, public and private detection tracks tab. 4

* SORT is run without the public-detection filter tab. 4 caption.

MOT17 and MOT20 private, as quoted in the text: TransMOT-D is 10.0 points above TransTrack in IDF1 and 3.0 points above it in MOTA §5.2. On MOT20 the private TransMOT is 6.6 points above CSTrack in IDF1 §5.2. Moving from the private to the public setting on MOT20, MOTA drops by 4.4 points and IDF1 by 0.9 §5.2.

Ablation on the MOT17 training set, detector trained on CrowdHuman only [tab. 5] §5.3:

Qualitative results are shown in fig. 5 (sequences from MOT15, 16, 17, 20).

Paper vs code

No official code. Searches: the arXiv abstract page (no code link); a GitHub repository search for "TransMOT" (results were unrelated projects with similar names); "TransMOT multiple object tracking" (no results); "TransMOT" limited to the microsoft organisation (no results); and "transmot tracking" (no results). The LaTeX source has no repository URL.

Pitfalls

• Loss sign (eq. 4). As typeset, only the first term carries a leading minus. The two sink terms are written as log-probabilities with a plus, so the sum as printed is not a loss to minimise. The paper does not say whether the sink terms are meant to be negated. The LaTeX source contains a commented-out author note asking whether the minus on the sink term should be removed (source line 282; this note is not in the published text). Decide the sign by a sanity check on a toy case, and document the choice.
• Sparsity claim vs equation. The paper says the graph weights make attention non-zero only for interacting tracklets §4.1.1. Eq. 3 multiplies the scores by the weights before the softmax. For a pair with zero IoU the score becomes zero, which is not -inf, so such a pair still gets softmax mass unless the implementation masks it. The paper does not state the masking. Check the code before assuming sparsity.
• Decoder tensor shapes. The text gives shapes (M+1)x1xD, (M+1)x(N+1)xD and Tx(N+1)xD but does not say how the T axis of the tracklet embedding enters the cross attention §4.2. Choose a pooling or attend-over-T scheme and document it.
• Relaxed assignment. Training allows one track to receive several detections. Inference then needs a second matching step (stage 2 in §4.4), whose algorithm is not named there and whose threshold is not stated.
• Virtual sink targets. The sink row is trained as a multi-label target, but how y_n^snk is derived from ground truth (exit versus occlusion) is not stated §4.3.
• Unstated thresholds. Stage 1 and stage 2 thresholds, the stage 3 cost weighting between feature distance and D_top, the duplicate-removal threshold, and the Kalman parameters are not stated §4.4.
• Hidden costs. The full tracker runs at 9.6 fps, with the detector alone at 15.4 fps §5.1. The detector is a YOLOv5 variant trained for 300 epochs on CrowdHuman plus MOT training sets; those training costs are not included in the reported speed §5.1.
• Ablation vs main results. Ablations use a CrowdHuman-only detector and the MOT17 training set §5.3. Their numbers are not comparable with the test-set tables.
• "State of the art on all datasets". The abstract claims it, but on MOT15 the text claims SOTA only for IDF1, MT, FN and IDS §5.2; in Table 1 GSDT has higher MOTA. The text attributes low MOTA on MOT15 to incomplete annotation §5.2.
• Typos in the text. Section 5.3 writes L where it means T ("increasing L beyond 5"). Use T.
• Unnumbered formulas. The D_top formula and the multi-head output formula in §4.1 and §4.4 carry no equation number; cite by section.

Lineage

• Builds on: the Transformer attention mechanism §4.1 (cited as the scaled dot-product of Vaswani et al.), the ChebConv graph convolution §5.1, and the graph-based MOT tradition of multi-cut and lifted-edge formulations §2.
• Compared against: TransTrack and TrackFormer, the transformer-based MOT trackers of the same period §2 tab. 3; the graph-based MPNTrack and LifT tab. 3; and the detection-based trackers in Tables 1 to 4.
• Superseded by: not stated in this wiki.