MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking

In plain words

Following several people at once in a video is hard when they look alike and move in erratic ways. Most recent trackers that work end to end only remember what a person looked like in the previous frame or two. If the same person is hard to see for a while, the tracker can lose track of who is who.

MeMOTR adds a memory for each tracked person. The memory changes slowly, so it is a steadier summary of that person than the last frame alone. Before the tracker decides where each person is now, it blends this memory into the person's internal representation. A second step lets the people in the same frame compare themselves to each other, so each one stays distinct from its look-alikes. The authors also split the first decoder step off to find new people, so that finding newcomers and following known people do not compete as much.

The result is aimed at association: keeping the right identity on the right person over time. The authors report the largest gains on crowded dance videos, where people look alike and move unpredictably.

TL;DR

MeMOTR is a Transformer tracker that maintains one long-term memory vector per tracked object, updated as an exponential running average of the object's output eq. 1. The memory is injected into the track embedding through a memory-attention layer §3.4. On DanceTrack the authors report 68.5 HOTA and 58.4 AssA on the test set, against 54.2 and 40.2 for MOTR tab. 1. Use it if association on crowded or irregular motion is the bottleneck. Expect weaker detection than the best detector-based trackers. The official code is MIT licensed and the paper's training schedule differs from the code in places (see Paper vs code).

Problem

Tracking by detection links per-frame boxes with appearance and linear motion cues, which fail when motion is irregular or many people look alike §1 §2. Transformer tracking-by-query methods such as MOTR track objects end to end, but they mainly use the states of objects in the two adjacent frames §1 §2. The authors argue that this leaves out long-term temporal information, which matters when appearances are similar, motion is erratic, or a person is occluded §1. They also argue that DETR's learnable detect query has no object-specific meaning, while a track query does, and that mixing the two in one decoder causes misalignment §3.2.

The paper's main benchmark, DanceTrack, is chosen because it stresses association more than pedestrian benchmarks do §4.1. Its results therefore show the largest gains on association metrics and smaller gains on detection tab. 1.

Method

Overview §3.1 fig. 1. A ResNet-50 backbone and a Transformer encoder produce image features for each frame, following DETR-family detectors. The model has two kinds of query. Detect queries are learnable and find newborn objects. Track queries carry each tracked object forward. The decoder is split: one detection-only layer runs on the detect queries §3.2, and the remaining layers jointly process detect and track embeddings §3.2. Each tracked object also has a long-term memory M, which is injected into its track embedding by the temporal interaction module (TIM) §3.3 §3.4.

Detection decoder §3.2. The first decoder layer uses only the learnable detect query and produces a detect embedding. The detect embedding is then concatenated with the track embeddings for the joint layers. The authors report that a single detection-only layer gives the best association result among the depths they tried tab. 4.

Long-term memory §3.3. A newborn object's memory is initialised from its current output §3.3. After each frame, the memory is updated as an exponential running average with rate λ, set to 0.01 in the experiments eq. 1 tab. 7:

\widetilde{M}_{tck}^{t+1} = (1 - \lambda)\, M_{tck}^{t} + \lambda\, O_{tck}^{t} eq. 1

Temporal interaction module (TIM) §3.4 fig. 2. The TIM runs in four steps for each tracked object.

1. A channel-wise confidence weight is computed from the current output as a sigmoid of a small MLP applied to it eq. 2. W_{tck}^{t} = \mathrm{Sigmoid}(\mathrm{MLP}(O_{tck}^{t})) eq. 2
2. Adaptive aggregation: the weighted current output is concatenated with the previous frame's output, and a two-layer MLP fuses them. The previous output is not weighted §3.4 tab. 5.
3. Memory attention: a multi-head attention layer uses the aggregated feature as query, the long-term memory as key, and the current output as value §3.4.
4. The attention result is added to the long-term memory and passed through an FFN to give the next track embedding §3.4. The memory itself is updated by eq. 1.

Inference and updating §3.5. A detection with confidence above τ_det = 0.5 becomes a newborn track. A tracked object whose confidence is at or below τ_track = 0.5 is kept as inactive and removed after T_miss frames. T_miss is 30 on DanceTrack, 15 on MOT17 and 10 on BDD100K §3.5. Only objects with confidence above τ_next = 0.5 have their track embedding and memory updated eq. 3. The others keep their previous values.

[E^{t+1}_i, M^{t+1}_i] = \begin{cases} [\widetilde{E}^{t+1}_i, \widetilde{M}^{t+1}_i], & c_i^t > \tau_{next} \\ [E^{t}_i, M^{t}_i], & c_i^t \leq \tau_{next} \end{cases} eq. 3

Training §4.2. The model is built on DAB-Deformable-DETR with a ResNet-50 backbone, initialised from COCO-pretrained DAB-Deformable-DETR weights. A clip of 2 to 5 consecutive frames is sampled with a random interval between 1 and 10 frames. The longest clip length grows over training (schedule in Hyperparameters). Training uses AdamW with an initial learning rate of 2.0e-4, batch size 1 per GPU, and 8 V100 GPUs. A gradient-checkpointed version is described as fitting within 10 GB §4.2.

Pseudocode for one tracked object, from the paper's description. It is a paraphrase of §3.4 eq. 1 eq. 2 eq. 3, not the released code. See Paper vs code for the differences.

# Temporal interaction module for one track  [§3.4]
# Input:  out (C,) current output; out_prev (C,) previous output;
#         memory (C,) long-term memory; embed (C,) track embedding;
#         score (scalar) current confidence
# Output: next track embedding and next memory

def temporal_interaction(out, out_prev, memory, embed, score):
    # 1. Channel-wise confidence weight from the current output  [eq. 2]
    weight = sigmoid(mlp_weight(out))
    # 2. Adaptive aggregation over current and previous outputs  [§3.4]
    fused = mlp_fuse(concat(weight * out, out_prev))
    # 3. Memory attention: query fused, key memory, value out  [§3.4]
    attended = memory_attention(query=fused, key=memory, value=out)
    # 4. Add memory to the attention result, then apply an FFN  [§3.4]
    next_embed = ffn(memory + attended)
    # 5. Exponential running average of the output, lambda = 0.01  [eq. 1]
    next_memory = (1 - lam) * memory + lam * out
    # 6. Update only when the score is above 0.5  [eq. 3]
    if score > 0.5:
        return next_embed, next_memory
    return embed, memory

# Newborn track start  [§3.3]
# Input:  out_det (C,) output of the detection for the new object
# Output: initial long-term memory

def start_track(out_det):
    # The memory starts from the current output  [§3.3]
    memory = out_det
    return memory

Hyperparameters

Values are as stated in the paper unless marked "code". Code values are from the official repository at code_commit and are the training configs shipped there.

Results

Author-reported. The setup for every table is given in §4.1: DanceTrack train and test sets, MOT17 with CrowdHuman, BDD100K val, and SportsMOT. Gray rows in the paper use extra training data and are listed for context only.

DanceTrack test tab. 1. Caption: existing methods' results are from the DanceTrack paper. MeMOTR* is the Deformable-DETR variant.

MOT17 test tab. 2. Caption: the best Transformer-based result is in bold. MOTRv2 is listed as hybrid because it uses YOLOX proposals.

BDD100K val tab. 3. Caption: existing methods' results are from TETer. The text says MOTR was re-evaluated on the new metrics with its official model (see Pitfalls).

SportsMOT test tab. 10 app. D. Gray rows use extra data. The caption says ByteTrack and OC-SORT were run with their official code and trained only on the SportsMOT training set.

Ablations on DanceTrack val (trained on train, evaluated on val §4.6).

Detection decoder depth, L_det = number of detection-only layers tab. 4:

Adaptive aggregation tab. 5. Columns: O_{t-1} = previous output used; W = channel-wise weight used.

Long-term memory and memory-attention tab. 6. "naïve" is a single FFN on O_t without aggregation or memory.

Memory update rate λ tab. 7:

Supplemental DanceTrack ablation tab. 8 app. A. Columns: val (left) and test (right). "D-D-DETR" is Deformable-DETR; "DAB-D-DETR" is DAB-Deformable-DETR. Rows marked gray use an extra detector.

Difficult DanceTrack val sequences tab. 9 app. B. The eight sequences with the lowest MOTR AssA. MOTR results come from the official checkpoint.

The paper's Figure 3 visualises anchors before and after the detection decoder fig. 3. Figure 4 visualises t-SNE of track embeddings for the four combinations of memory and memory-attention fig. 4. Appendix app. C extends this to two more sequences fig. 5. These are qualitative only.

Paper vs code

Repository: https://github.com/MCG-NJU/MeMOTR, MIT license, commit eb7a177b9cbcb89742ec69b2545ab3af2ea31a80. The repository is linked from the paper's abstract, and its LICENSE names the Multimedia Computing Group, Nanjing University. Each difference below was checked against the code.

• DanceTrack epochs. The paper trains 18 epochs §4.2. The shipped config sets 20 train_dancetrack.yaml and says it changed some parameters from the paper for more stable convergence [code: configs/train_dancetrack.yaml#L3]. The saved run config also has 20 [code: outputs/memotr_dancetrack/train/config.yaml#L23].
• BDD100K epochs and interval. The paper trains 14 epochs §4.2. The config sets 13 with the comment "or 14, is OK" [code: configs/train_bdd100k.yaml#L72]. The paper says nothing about the interval, and the config uses a fixed interval of 4 [code: configs/train_bdd100k.yaml#L69].
• Newborn memory. The paper initialises the memory from the current output §3.3. The code stores the query embedding in the newborn's query_embed and uses that as memory query_updater.py [code: models/criterion.py#L333-L334]. For the DAB model, query_embed is aux_outputs[-1]["queries"]. set_aux_loss pairs the first five layers with queries[1:] memotr.py, and the query tensor is appended before each layer runs deformable_decoder.py. That makes the newborn memory the state entering the last decoder layer, not the final output_embed (outputs[-1]) memotr.py. This is a different tensor from the one the paper names.
• Memory-attention positional term. The paper describes the query, key and value without a positional term. The code adds a sine embedding of each track's reference box, passed through query_pos_head, to both the query and the key query_updater.py [code: models/query_updater.py#L120-L125].
• Order of memory and attention. The paper adds the memory to the attention result and then applies one FFN §3.4. The code applies attention with a residual on output_embed, a LayerNorm and an FFN, then adds the memory, a LayerNorm and a second FFN query_updater.py.
• Reference boxes. The code overwrites the reference point of each confident track with its current predicted box before computing the positional term query_updater.py. The paper does not describe this.
• Filter during training. The paper says tracked objects below score 0.5 and IoU 0.5 are filtered out during training §4.2. The shipped configs set both augmentation rates to 0, so the code takes the branch that keeps every tracked ID and sets the ID to -1 when IoU is below 0.5 [code: models/query_updater.py#L186-L192]. The dropping filter, with IoU above 0.5 and ID non-negative, runs only when a rate is above 0 [code: models/query_updater.py#L194-L195].
• Input size. The paper resizes the shorter side to 800 with max 1536 §4.2. Training in the code samples a shorter side from 608 to 992 with max 1536, or resizes to 800 to 1200 and crops dancetrack.py. Evaluation uses 800 with max 1333 dancetrack.py. Horizontal flip and HSV jitter are in the code only [code: data/dancetrack.py#L155] dancetrack.py.
• Optimiser and loss. AdamW with weight decay 5e-4, backbone LR 2e-5, sampling-offset LR 1e-5 (DanceTrack) or 2e-5 (MOT17), gradient-norm clip 0.1, matching and loss weights 2/5/2, aux weight 1 per layer. The paper does not state these train_engine.py train_engine.py [code: configs/train_dancetrack.yaml#L94-L111] train_mot17.yaml. The clip call uses the literal 0.1, not the configured value [code: train_engine.py#L242].
• Learning-rate milestone. The paper says "decay at the 12th epoch". The config passes [12] to a step scheduler stepped once per epoch [code: train_engine.py#L57-L60] train_engine.py. Decay therefore starts at 0-based epoch 12, which is the 13th epoch counting from 1.
• Query-updater-only phase. ONLY_TRAIN_QUERY_UPDATER_AFTER zeroes the learning rate of the backbone, sampling points and query updater from that epoch on train_engine.py. It is set equal to EPOCHS for DanceTrack, SportsMOT and MOT17, and to 1000 for BDD100K, so it is inactive in every shipped config.
• Track placeholder. When no track survives the filter, the code inserts a placeholder with random embeddings and ID -2 [code: models/query_updater.py#L219-L241].
• Inference thresholds. The runtime tracker's defaults (0.7, 0.6, miss 5) differ from the configs (0.5, 0.5, 30). The submit path reads the config values [code: models/runtime_tracker.py#L14-L15] submit_engine.py.
• Pseudo-trajectories for MOT17. The paper describes CenterTrack-style random shifts of CrowdHuman images to make pseudo trajectories §4.2. This was not checked in the code; the loader's CrowdHuman handling is at [code: data/mot17.py#L36-L37] and mot17.py.

No discrepancies were found in the core memory update rule eq. 1, the confidence weight eq. 2, the update rule eq. 3, the detection-only first layer §3.2, λ = 0.01 §3.3, or the DanceTrack clip-length schedule §4.2.

Pitfalls

• Paper-internal inconsistencies. Four statements in the paper do not match its own tables:
  • λ ablation: the text says DetA decreases as λ grows §4.6. Table 7 shows DetA rising from 74.2 to 75.1 tab. 7.
  • Adaptive aggregation: the text gives +2.6 HOTA for using both O_{t-1} and W §4.6. Table 5 shows 63.9 against 62.3 with neither, a gap of 1.6, and 63.9 against 62.4 with W only, a gap of 1.5 tab. 5. The +2.3 AssA claim matches the table.
  • DanceTrack test MOTA is 89.9 in Table 1 tab. 1 and 89.8 in Table 8, row 6 tab. 8.
  • BDD100K: the caption says the existing methods are from TETer tab. 3, while the text says MOTR was re-evaluated with its official model §4.5.
• Abstract figures are absolute points. The abstract's "7.9%" and "13.0%" are differences in points to C-BIoU's Table 1 values (60.6 HOTA, 45.4 AssA) [abstract] tab. 1. They are not relative gains.
• Extra-data baselines. MOTRv2, which uses extra data, reports 69.9 HOTA against MeMOTR's 68.5 tab. 1. The headline "state of the art" claim is against methods without extra data.
• Detection trade-off. MeMOTR's DetA on DanceTrack is 80.5 against 81.3 for C-BIoU tab. 1. The authors note weaker detection in crowds §4.7.
• Detector dependence. The Deformable-DETR variant reaches 63.4 HOTA, against 68.5 with DAB-Deformable-DETR tab. 1 tab. 8. Part of the gain is the detector, not the memory tab. 8.
• Epoch count versus paper. Use the config values (20 for DanceTrack) if you reproduce the released code, and the paper's 18 if you follow the text.
• Memory is per track, constant size. Memory is one vector per track, so storage does not grow with clip length §3.3. The compute cost is the per-frame sequential loop with an attention over tracks; the paper's gradient-checkpointed build is what fits in 10 GB §4.2.
• Random clip interval. The interval is drawn uniformly from 1 to the stage maximum, capped by the frames left in the video dancetrack.py. A short video yields smaller intervals than the paper's wording suggests.
• Empty track set. Training inserts random embeddings when no track survives the filter query_updater.py. The paper does not mention this. This card did not trace whether the placeholder reaches the loss.
• Training-only filter. The IoU filter that removes tracked objects with low IoU is in the branch that the shipped configs do not use. Reproducing the paper's filter exactly means using the other branch.
• Memory-update threshold. The same 0.5 threshold gates both the track embedding and the memory eq. 3. A track below 0.5 keeps its previous embedding and memory unchanged eq. 3 until it is re-confirmed or retired §3.5. The paper does not say how stale a memory may get.

Lineage

• Builds on: DETR for end-to-end set prediction §3.2; Deformable DETR and DAB-Deformable-DETR for the detector, the latter with learnable anchors §3.1 §4.2 app. A; MOTR for the tracking-by-query paradigm §1 §2. The paper's bibliography has no arXiv identifiers for these, so they are not linked. The exponential running average in eq. 1 is attributed to an earlier work cited in §3.3.
• Compared against: MeMOT, a memory-bank tracker §2 tab. 2; MOTRv2, a hybrid method that adds YOLOX proposals §4.4 app. A; ByteTrack, OC-SORT, C-BIoU and TransTrack tab. 1 tab. 2.
• Superseded by: not stated.