Paper · 2024-08-23 · cs.CV
MCTR: Multi Camera Tracking Transformer
MCTR tracks objects across overlapping cameras with one transformer. A DETR detector runs per view; a set of track embeddings is updated from all views each frame and softly assigned to detections, so a pairwise same-track loss trains the whole system end to end. On MMPTrack and an AI City set it beats end-to-end MOTR run per camera, but trails top challenge systems on IDF1.
Visual object trackingObject queries (learned query slots in transformer decoders)Bipartite matching for set-prediction lossesCross-attentiontrackingmulti-cameratransformerdetrend-to-endassociation
arXivPDFCodeMarkdown13 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Following people or objects through a room with several cameras is hard. The same person has to keep one ID in every camera and over time. Most existing systems chain many hand-built parts together to do this.
MCTR replaces most of that chain with one neural network. A detector finds the objects in each camera separately. A shared set of "track slots" holds what the system knows about each tracked object. At every frame, those slots are updated with information from all cameras. Each slot is linked to the detections in each camera with soft probabilities, not hard choices. The training signal asks the model to give high probability to pairs of detections that show the same object. Because the links are soft, the whole system can be trained end to end.
On the benchmarks in the paper, MCTR beats an end-to-end single-camera tracker run on each camera. It still trails the best hand-tuned challenge systems, mainly on the identity score IDF1.
TL;DR
- Multi-camera tracking with overlapping views was mostly heuristic; MCTR is an end-to-end transformer for it §1.
- DETR runs per view; a set of track embeddings is updated from all views each frame and softly assigned to detections §3.
- The pairwise same-track loss, built from the soft assignments, makes the whole system trainable end to end §3.4.
- Reported: MCTR beats MOTR (per camera) and ReST on MMPTrack; it trails the top MMP-Tracking challenge entries on IDF1 §4.1, tab. 2.
- The paper has no implementation section in the published source; training settings come from the official code.
Problem
Tracking across cameras whose views overlap combines re-identification, single-camera tracking, homography and clustering. Those parts need hand-set rules and their results depend on how well those rules are tuned §1. End-to-end transformer trackers exist for a single camera. Single-camera methods such as MOTR use track embeddings as detection queries. That assumes one track has at most one detection per time step, which breaks when several cameras see the same object at once §1. A plain extension is therefore unclear, both for keeping IDs consistent across views and time and for writing a loss that trains the system end to end §1.
Method
MCTR is built from three parts §3: detection, tracking and association. fig. 2 gives an overview.
Detection module
Each camera view goes through a vanilla DETR (backbone, transformer encoder and decoder). Each output embedding yields a class (or "no object") and a box. The paper says any end-to-end detector could be used §3.1. The output embeddings are passed to the tracking and association modules whether or not they predict an object §3.1 fig. 3.
Tracking module
The module keeps a set of track embeddings meant to hold global information about each object §3.2 fig. 4.
- At the first frame the track embeddings start from learned track query embeddings. A track that stays unassigned for a while is reset to its initial embedding §3.2.
- Each camera view has its own multi-head cross-attention. Track embeddings are the queries, and that view's detection embeddings are keys and values. Parameters are not shared across views §3.2.
- The per-view results are combined by a mean, then a self-attention block and a feed-forward block update the tracks §3.2. The self-attention lets tracks compete for identities, similar to DETR's decoder §3.2.
Association module
For each view, detections and tracks each pass through a linear transform. Their dot products are divided by √d, the width of the embeddings, and a softmax normalises each detection's scores across tracks. The output is a D×T matrix A^v; each cell estimates how likely a given detection in this view is linked to a given track. D counts detection queries and T counts track queries §3.3 fig. 5.
Training loss
The loss has three parts §3.4. The equations in §3 are unnumbered in the source.
- Detection loss. DETR's loss (classification negative log-likelihood, IoU and GIoU), with Hungarian matching per view and frame. Auxiliary losses are taken from intermediate decoder layers §3.4.
- Pairwise track loss. For two detections d1 and d2 in different views (same frame), or in the same view (different frames), the model's probability that they share a track is P(d1, d2) = Σ_t A^{v1}[d1, t] · A^{v2}[d2, t] §3.4. The label is 1 if the two matched ground-truth boxes share a track ID and 0 if they differ. The pair is skipped when either detection is unmatched §3.4. The loss is a binary cross-entropy averaged over the defined pairs §3.4.
- Auxiliary track box loss. A view-specific three-layer MLP predicts a box for each track in each view. The IoU and GIoU losses are weighted by A over matched detections §3.4.
The total loss is an unweighted sum of these terms §3.4.
Training protocol
The first thirty epochs use four-frame clips. This is meant to expose the detector to diverse data. After that the detector is frozen, and the tracking and association modules train on longer video segments §3.5. Segment length is drawn from a geometric distribution whose expected value grows linearly during training §3.5. Track embeddings carry over from the previous clip §3.5. Longer segments improve IDF1 and HOTA, while MOTA stays roughly flat; short segments do not give the same gain §3.5 fig. 6.
Inference
Inference is online, one frame at a time §3.6.
- In each view, keep detections whose confidence is above 0.9 §3.6.
- Run the Hungarian algorithm on A^v to find the matching that maximises total association probability §3.6.
- If a track has had no detection for more than four frames, reset its embedding to the initial embedding §3.6.
Cost: the single-camera build has roughly 44M parameters and needs 233 MB of memory; it runs at 21 FPS on an RTX 2080 Ti. Each extra camera adds about 1M parameters and 68 MB and lowers the speed by about 2 FPS, so time and memory grow linearly with cameras §3.6 fig. 7.
# One clip step: detect, update tracks, associate [§3.1] [§3.2] [§3.3]
# Input: frames (F, V, 3, H, W); tracks (T, d) from the previous clip
# Output: association A (F, V, D, T); updated tracks (T, d)
def mctr_clip_step(frames, tracks):
# Run DETR on each view and frame independently [§3.1]
dets = detr_decoder(frames) # (F, V, D, d)
for frame in range(F):
# Per-view cross-attention, then mean over views [§3.2]
updates = []
for view in range(V):
updates.append(cross_attn[view](tracks, dets[frame, view]))
averaged = mean(updates)
# Self-attention and feed-forward on the tracks [§3.2]
tracks = feed_forward(self_attention(averaged))
for view in range(V):
# Dot-product scores between detections and tracks [§3.3]
scores = proj_det[view](dets[frame, view])
scores = scores @ proj_trk[view](tracks).T
# Row-wise softmax over tracks, scaled by sqrt(d) [§3.3]
assoc[frame, view] = softmax(scores / sqrt(d), dim=-1)
return assoc, tracks# Training loss for one clip [§3.4]
# Input: detections per view and frame, association A, ground truth
# Output: scalar loss
def clip_loss(dets, assoc, targets):
# Hungarian matching, per view and frame [§3.4]
matches = hungarian_per_view(dets, targets)
# DETR loss on the matched pairs [§3.4]
loss = detr_loss(dets, targets, matches)
for d1, d2 in pairs_of_matched_detections:
# Probability that d1 and d2 share a track [§3.4]
p_same = sum(assoc[d1] * assoc[d2]) # sum over tracks t
# Label from the ground-truth track IDs [§3.4]
label = track_id(d1) == track_id(d2)
# Average over the N_p defined pairs [§3.4]
loss += binary_cross_entropy(p_same, label) / N_p
# Auxiliary track box loss (IoU and GIoU) [§3.4]
loss += track_box_loss(assoc, targets)
return loss# Online inference, one frame [§3.6]
# Input: detections and A per view; tracks; frames since each last match
# Output: track ID for each kept detection
def infer_frame(dets, assoc, tracks, frames_unused):
for view in range(V):
# Keep detections with confidence above 0.9 [§3.6]
keep = dets[view].confidence > 0.9
# Hungarian on A^v, maximising total association probability [§3.6]
matches = hungarian_maximise(assoc[view][keep])
frames_unused = update_unused(frames_unused, matches)
# Reset tracks unmatched for more than four frames [§3.6]
for track in tracks:
if frames_unused[track] > 4:
tracks[track] = initial_track_embedding # [§3.2]
return matchesHyperparameters
The paper's text gives almost no training settings: its implementation-details
subsection is commented out in the source (sec/4_results_new.tex, line 387).
Rows marked "code" come from the official repository at code_commit. Rows marked
"not stated" are not in the paper.
| Name | Value | Where | Notes |
|---|---|---|---|
| Detector | vanilla DETR, ResNet-50 backbone | §3.1; pairwise.yaml | Paper: "vanilla DETR"; backbone not stated in paper |
| Detector pretrained init | DETR R50 checkpoint detr-r50-e632da11.pth; class head cut to 2 outputs, queries cut to 20 | main_pairwise.py | Code only; the paper does not say how the detector is initialised |
| Detection queries per view | 20 | pairwise.yaml | Code only; not stated in paper |
| Track queries (track embeddings) | 20 | pairwise.yaml | Code only; not stated in paper |
| Tracking layers | 1 | pairwise.yaml | Code only; not stated in paper |
| Model width | 256 + 4×8 = 288 | config.py; pairwise_model.py | Detection embedding is concatenated with its predicted box (ADD_BBOXES) |
| Attention heads | 8 | config.py | Code only |
| Feed-forward width, dropout | 2048, 0.1 | config.py | Code only |
| Association scaling | divide by √d (d = 288 in code) | §3.3; pairwise_model.py | Paper gives the scaling; the value of d is not stated |
| Clip length | 4 frames | §3.5; pairwise.yaml | |
| Stage 1 length | 30 epochs, detector trained | §3.5; pairwise_init.yaml | |
| Stage 2 length | 100 epochs, detector frozen | pairwise.yaml, [#L124] | Paper gives no stage-2 length |
| Batch size | 32 | pairwise.yaml | Not stated in paper |
| Iterations per epoch | 600 | pairwise.yaml | Not stated in paper |
| Optimiser | AdamW, lr 1e-4, weight decay 0 | pairwise.yaml | Not stated in paper |
| LR for detector parameters | 1e-4 (same as rest, including backbone) | main_pairwise.py; pairwise.yaml | DETR's own backbone LR (1e-5) in config.py is not used |
| LR schedule | StepLR ×0.3 every 20 epochs; stepped per iteration | pairwise.yaml; main_pairwise.py; trainer.py | Not stated in paper |
| Gradient clipping | norm 1.0 (stage 1), 3.0 (stage 2) | pairwise_init.yaml; pairwise.yaml | Not stated in paper |
| Detection loss weights | CE 1, no-object weight 0.1, L1 box 5, GIoU 2 | config.py; pairwise_losses.py | Paper gives no weights |
| Pairwise loss weight | 5 | pairwise_losses.py | Paper: unweighted sum |
| Track-box loss weights | L1 2, GIoU 1 (also prev and next) | pairwise_losses.py | Only active when PRED_TRACK_BOXES is True |
| Top-down loss weight | 4 | pairwise_losses.py | Stage 1 only |
| Confidence threshold | 0.9 | §3.6; pairwise_postprocessor.py | Matches paper |
| Unused-track reset | > 4 frames | §3.6; trackeval_mmptrack_reset1.py | Main eval script uses 3 clips = 12 frames; see Paper vs code |
| Input frame size | 360 × 640 | pairwise.yaml | AI City: 640 × 360 §4.2; shared config uses the same shape |
| Augmentation | one of five photometric transforms per clip | mmptracking.py | Not in paper |
| Epochs for AI City | 50 (stage 1), 200 iterations per epoch | pairwise_init.yaml | Not in paper |
Results
Author-reported. The paper reports headline results on MMPTrack (five environments) and on a 2023 AI City Challenge Track 1 set §4. Metrics are HOTA, IDF1 and MOTA per camera view and test clip, averaged per environment §4. Cross-view metrics are AIDF1 and MOAA §4. The MOTR baseline is run per camera with its default parameters and with NMS at IoU 0.95 added §4.1. ReST is a multi-stage baseline; with ground-truth boxes it is called ReST-GT §4.1. The ReST and ReST-GT runs did not finish on some environments §4.1.
Ablation, industry environment tab. 1. The paper says this is the industry environment §4.1.
| Model | HOTA | IDF1 | MOTA | AIDF1 | MOAA |
|---|---|---|---|---|---|
| MCTR (with auxiliary losses) | 71.81 | 80.21 | 91.19 | 94.52 | 90.39 |
| MCTR-noaux | 68.00 | 75.26 | 92.11 | 95.00 | 91.30 |
MMP-Tracking challenge comparison, camera-view evaluation tab. 2. The paper evaluates on the validation set, not the test set, and assumes the two are comparable §4.1. The rank MCTR would have achieved is shown in parentheses.
| Model | IDF1 | MOTA | Online | Camera parameters | FPS |
|---|---|---|---|---|---|
| Hikvision | 86.36 | 87 | No | Yes | - |
| Alibaba | 88.18 | 78 | No | Yes | < 1 |
| MCTR | 62.30 (5) | 81.01 (3) | Yes | No | 9 |
The table's FPS setup (number of cameras, hardware) is not stated tab. 2.
Other reported figures in the text:
- MMPTrack: the paper reports MCTR ahead of both ReST and MOTR in every environment and on every metric. Against ReST-GT, MCTR is better on HOTA and IDF1, while ReST-GT is better on MOTA, MOAA and AIDF1 §4.1 fig. 8.
- Retail with track-level boxes (MCTR-TB): recall rises from 54.28 to 88.73. The paper credits track-level boxes that predict people the view detector misses, although the model never saw labels in the occluded regions §4.1.
- AI City: the paper uses the first 70% of each scene for training and the rest for testing, and resamples to 640 × 360 at 15 FPS §4.2. MCTR beats MOTR on IDF1 and HOTA; MOTR is higher on MOTA, which the paper attributes to deformable DETR and to many small, occluded people §4.2 fig. 9.
The per-environment and per-scene bar heights are in fig. 8 and fig. 9; they are not printed as numbers in the figures.
Paper vs code
Checked against the official repository, LICENSE BSD-3-Clause, at
a6b20765ebfaa56daaae9566e4fb5a266903ed8e.
- Box regression. Paper: IoU and GIoU losses §3.4. Code: L1 box loss (weight 5) and GIoU (weight 2) pairwise_losses.py pairwise_losses.py.
- Track-box loss. Paper: IoU and GIoU, normalised by 1/(V·T) §3.4. Code: L1 and GIoU, weighted by the assignment probabilities, normalised by the number of boxes, and weighted 2 and 1 pairwise_losses.py pairwise_losses.py pairwise_losses.py.
- Total loss. Paper: unweighted sum §3.4. Code: weights of 1 (classification), 5 (L1), 2 (GIoU), 5 (pairwise), 4 (top-down, stage 1) and the track-box weights pairwise_losses.py.
- Pairwise normalisation. Paper: one average over the N_p defined pairs §3.4. Code: a mean over pairs for each frame (cross-view), then averaged over frames; and for cross-frame, a mean per view, averaged over views pairwise_losses.py.
- Track initialisation. Paper: learned track query embeddings §3.2. Code: track content starts at zero; the learned query embedding is added only as a positional term on the first frame of a clip for reset slots (ZERO_QE) pairwise_model.py pairwise_model.py pairwise_model.py.
- Stage 1 clips. Paper: non-overlapping four-frame clips §3.5. Stage 1 config uses SUBSAMPLE_RATE 1, which creates a new sample every frame, so clips overlap pairwise_init.yaml. Stage 2 uses 4 pairwise.yaml. Stage 1 also resets every clip, because RESET_TGT is True pairwise_init.yaml.
- Unused-track reset. Paper: reset after more than four frames §3.6. The MMPTrack configs set RESET_UNUSED to 3 clips, which is 12 frames in the main eval script pairwise.yaml trackeval_mmptrack.py. The
reset1script uses 1 clip, which is 4 frames, and its comment says it was changed from 3 to 1 trackeval_mmptrack_reset1.py. - Track-box head on MMPTrack. The paper's MCTR-TB needs track boxes §4.1. The MMPTrack configs set PRED_TRACK_BOXES to False, with the comment "NOTE: changed for ablation" pairwise.yaml pairwise_init.yaml. The AI City config sets it to True pairwise.yaml. The MMPTrack track-box setup is therefore not in the shipped MMPTrack configs.
- Association input. Paper: linear transforms of detection and track embeddings §3.3. Code: the detection embedding is concatenated with its predicted box before projection pairwise_model.py pairwise_model.py.
- Stage 1 extras. Code: a top-down head is trained in stage 1 (PRED_TOPDOWN True, weight 4) pairwise_init.yaml pairwise_losses.py. The paper does not mention it.
- Detector initialisation. Code: DETR R50 checkpoint loaded, class head reduced to person and no-object, queries cut to 20 main_pairwise.py. Paper: not stated.
- Retail occlusion filter. Paper: YOLOX detections, IoU threshold 0.01 §4.1. Code: a script uses YOLO7 detections (feature file named
dets-yolo7), IoU threshold 0.01 generate_fixed_retail.py generate_fixed_retail.py. Its docstring warns it can drop boxes that YOLO does not detect generate_fixed_retail.py. - Clip gradient. Code: track embeddings are detached when passed to the next clip, so no gradient flows across clip boundaries pairwise_model.py. Paper: not stated.
Pitfalls
- No implementation section in the published source. Training settings (batch size, LR, epochs, queries, width) appear only in the code. Use the Hyperparameters table, and treat the code as the reference for anything it sets.
- Equations are unnumbered. The equations in §3 have no numbers in the source, so cite the section §3.4, not an equation number.
- Track count is fixed. Both configs use 20 track slots. The paper does not state a count, and some slots stay unassigned by design §3.2.
- Gradients stop at clip boundaries. Track embeddings are detached when they are passed to the next clip pairwise_model.py, even inside long stage-2 segments. Each gradient step therefore covers one four-frame clip; the carried track state is a constant.
- Markovian tracking. Track embeddings see only the current and previous frame. The authors say this may explain the gap in IDF1, and that longer temporal history did not help in early tries §5.
- Camera-specific weights. Each view has its own cross-attention. Adding a camera adds parameters (about 1M each) §3.6. The paper says the model depends on the camera setup it was trained on §5.
- Learning rate reaches the backbone. Detector parameters, including the backbone, use the same LR as the rest (1e-4), not DETR's default backbone LR of 1e-5 main_pairwise.py config.py.
- Stage-1 vs stage-2 clipping. Stage 1 uses overlapping windows (SUBSAMPLE_RATE 1), stage 2 does not (SUBSAMPLE_RATE 4). Check the config before comparing clip statistics pairwise_init.yaml.
- Baseline fairness. MOTR is given NMS at IoU 0.95 to remove duplicate boxes, which the paper says were a failure mode of MOTR with default settings §4.1. MCTR is not given this step.
- Annotation filter on retail. Ground-truth boxes with IoU below 0.01 against a YOLOX detection are removed for training. Evaluation uses unfiltered labels, and recall there is expected to be low §4.1.
- Comparison with challenge entries. Table 2 compares a validation-set score with challenge results. The authors assume validation and test are comparable §4.1. The FPS setup is not stated tab. 2.
- Cost scales with cameras. Time and memory grow linearly with the number of views, and each view costs about 2 FPS §3.6 fig. 7.
- Retail MCTR-TB. The track-level box recall gain relies on track boxes that the shipped MMPTrack configs disable (see Paper vs code, item 8) §4.1.
Lineage
- Builds on: DETR (Carion et al., 2020) for the per-view detector and its set-matching loss §3.1 §1.
- Differs from MOTR (Zeng et al., 2022), which uses track embeddings as detection queries; MCTR keeps separate track embeddings §1.
- Compared against: MOTR (Zeng et al., 2022), run per camera §4; ReST (Cheng et al., 2023), a multi-stage multi-camera tracker §4; MMP-Tracking challenge entries, Hikvision and Alibaba tab. 2.
- Superseded by: not known from content already in this wiki.
Source and license
Adapted from “MCTR: Multi Camera Tracking Transformer” by Alexandru Niculescu-Mizil, Deep Patel, Iain Melvin (arXiv:2408.13243), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0. Code citations refer to https://github.com/necla-ml/mctr at commit a6b20765ebfa (BSD-3-Clause). Any quoted code is shown under that license.