MeMOT: Multi-Object Tracking with Memory

In plain words

Tracking several moving things in a video means knowing, frame after frame, which box belongs to which earlier object. Many systems find boxes in each frame and then link them by comparing only the last frame or two. That breaks easily when objects are hidden for a while or cross each other.

MeMOT keeps a longer notebook for every tracked object: a short history of what it looked like and where it was. When a new frame arrives, the model looks back through that notebook and pulls out the useful parts, with one lookup for recent frames and another for older ones. The same network then finds new objects, updates the known ones and decides which boxes are real, so there is no separate matching step afterwards.

The authors judge the method by identity switches, where the tracker swaps the label of one object for another, and report fewer of them than several baselines on the public benchmarks they used.

TL;DR

MeMOT is an online multi-object tracker built from three Transformer modules: one proposes object boxes in the current frame, one compresses each track's stored history into one embedding, and one decodes boxes and identities together §3.1. Each track's history sits in a fixed-length first-in-first-out memory §3.3. The paper reports competitive results on MOT16, MOT17 and MOT20. Its identity-switch counts beat several Transformer baselines but not MOTR tab. 1. Results are author-reported.

Problem

Online tracking must link each new detection to an existing identity using only past frames. Methods that pass state only between adjacent frames lose an object that is occluded for longer than that window §1 §4.5. Methods that separate detection from association need a hand-built linking step after detection §4.3. Storing a long history for every object costs GPU memory during training, and the authors name this as the limiting factor §4.6 app..

Method

Overview. At frame t the model produces proposal embeddings for new objects, track embeddings for known objects, and then decodes both into boxes, objectness and uniqueness scores §3.1. The track embeddings come from the memory, which is updated after each frame §3.1 alg. 1.

Hypothesis generation \Theta_H §3.2. A CNN backbone turns the frame into features. A Transformer encoder with position encodings processes them, and a Transformer decoder maps a set of learned object queries to N^t_{pro} proposal embeddings. Objectness and boxes are predicted from these. The paper's implementation starts from ResNet-50 and Deformable DETR weights pretrained on COCO §4.2.

Spatio-temporal memory §3.3. For each tracked object the model stores its past states. The buffer holds at most N_{max} objects and T_{max} time steps per object, is first-in-first-out, and pads with zeros for frames where an object is absent. When a stored state is older than T_{max}, it leaves the memory. The text says T_{max}=24 §3.3; see Pitfalls for a conflict with §4.2.

Memory encoding \Theta_E §3.4 fig. 2. Three attention blocks produce one track embedding per track:

1. A short-term block f_{short} takes the most recent state as its query and the last T_s states as keys and values. It smooths noise in nearby frames.
2. A long-term block f_{long} takes a learnable "dynamic memory aggregation token" (DMAT) as its query and the last T_l states as keys and values. The DMAT is initialised once and then updated at every step.
3. A fusion block combines the two outputs into the track embedding and an updated DMAT, which is kept for the next step §3.4 alg. 1.

Both branches are multi-head cross-attention §3.4. The appendix describes the fusion as a self-attention module app.; the main text calls it a fusion block §3.4.

Memory decoding \Theta_D §3.5. Stacked Transformer decoder layers take the concatenation of proposal and track embeddings as queries and the frame features from \Theta_H as keys and values. Each output entry gets three predictions: a box, as offsets from learned reference points; an objectness score o \in [0,1]; and a uniqueness score u \in [0,1]. A track entry is given uniqueness 1; a proposal entry is supervised to be unique only when it matches no tracked object. The confidence score is their product eq. 1:

s = o \cdot u eq. 1

Entries with s above a threshold are kept. Track entries extend their track, and proposal entries start a new one. No further post-processing is applied §3.5 alg. 1.

Training §3.5 §3.6. Ground truth for track entries comes from the track's presence in the frame. Proposal entries are matched to ground-truth boxes by bipartite matching, whether or not the object is already tracked, and take their uniqueness label from whether the matched object was seen before §3.5 fig. 3. The tracking loss per entry is eq. 2:

L_{tck} = \lambda_{cls}(L'_{obj} + L'_{uni}) + \lambda_{L_1} L'_{bbox} + \lambda_{iou} L'_{iou} eq. 2

Here L'_{obj} and L'_{uni} are focal losses on objectness and uniqueness, L'_{bbox} is an L1 box loss and L'_{iou} a generalised IoU loss §3.6.

An auxiliary detection loss on the proposal embeddings follows Deformable DETR §3.6 eq. 3:

L_{det} = \lambda_{cls} L_{obj} + \lambda_{L_1} L_{bbox} + \lambda_{iou} L_{iou} eq. 3

It uses a separate linear head, discarded after training §3.6.

Over a clip of T frames, the tracking loss is summed over entries and divided by the total number of visible objects. The detection loss is averaged per frame and then over frames eq. 4 §3.6:

L_{clip} = \lambda_{tck} L_{clip\text{-}tck} + \lambda_{det} L_{clip\text{-}det} eq. 4

Training uses clips, following MOTR §3.6 §4.2.

# MeMOT online inference  [alg. 1]
# frames: a video; memory: per track, a FIFO of past embeddings

def track_video(frames):
    memory = empty_memory()                              # X <- empty  [alg. 1]
    for frame in frames:                                 # for I^t  [alg. 1]
        # Frame-level proposals  [§3.2] [alg. 1]
        features = hyp_encoder(frame)                    # z_1^t, (d, HW)
        proposals = hyp_decoder(features)                # Q_pro^t, (N_pro, d)

        # Memory encoding: one embedding per track  [§3.4] [alg. 1]
        short = f_short(memory.latest(), memory.last(T_s))
        long = f_long(dmat, memory.last(T_l))
        tracks, dmat = f_fuse(short, long)               # Q_tck^t, next DMAT
        # (DMAT is updated here, not once per clip)     [§3.4] [tab. 7]

        # Memory decoding and scoring  [§3.5] [eq. 1]
        props_out, tracks_out = decoder(
            concat(proposals, tracks), features)
        boxes_p, scores_p = predict(props_out)           # s = o * u
        boxes_t, scores_t = predict(tracks_out)

        # Extend tracks above the track threshold  [alg. 1] [app.]
        for k in range(len(tracks)):
            if scores_t[k] >= EPS_TCK:                   # 0.6  [app.]
                extend_track(k, boxes_t[k])
                memory.append(k, tracks_out[k])

        # Start new tracks above the proposal threshold  [alg. 1] [app.]
        for k in range(len(proposals)):
            if scores_p[k] >= EPS_PRO:                   # 0.7  [app.]
                start_track(boxes_p[k])
                memory.add(props_out[k])
    return trajectories()

# Clip training loss  [eq. 2] [eq. 3] [eq. 4]
# Per frame: track entries, proposal entries and visible-object count N_t

def clip_loss(frames, lam_tck=1.0, lam_det=1.0):
    tck_sum = 0.0
    det_sum = 0.0
    for out in frames:                                   # t = 0..T
        # Tracking loss over track and proposal entries  [eq. 2]
        for entry in out.entries:
            tck_sum += tracking_loss(entry)
        # Auxiliary detection loss over proposal entries  [eq. 3]
        det_sum += (1.0 / out.n_visible) * sum(
            detection_loss(j) for j in out.proposals)
    n_total = sum(out.n_visible for out in frames)
    loss_tck = lam_tck * tck_sum / n_total               # [eq. 4]
    # Eq. 4 divides by T, the last frame index (frames run t = 0..T)
    loss_det = lam_det * det_sum / (len(frames) - 1)     # [eq. 4]
    return loss_tck + loss_det

Hyperparameters

Values are as stated in the paper; "not stated" means the paper does not give them.

Results

Author-reported. Setup: MOT Challenge benchmarks, evaluation with CLEAR MOT metrics and HOTA §4.1. Training data is listed in the table as multiples of the MOT training set §4.2. Shaded (in-network association) baselines are the fair comparison §4.3. Full benchmark table: tab. 1.

The paper states that on MOT20 it has a 63% reduction in identity switches relative to FairMOT and is 6.7 IDF1 and 5.2 HOTA ahead of the other Transformer trackers §4.3. These comparisons match the table.

Ablations (on MOT15 validation, two-layer model; not comparable to the table above) §4.5:

Qualitative results, tracks on MOT17 and MOT20 fig. 4, and visualised attention for two identities through an occlusion fig. 5 are described in §4.4 only.

Paper vs code

No official code.

Searched for an official repository in this order:

• The LaTeX source (raw/src/2203.16761) contains no code or project URL. The only URLs in it are in the CVPR template header.
• The arXiv abstract page has no code link. Its comment field says only "CVPR 2022 Oral".
• GitHub repository search for memot returned 3,090 results. The first 19 shown contained no repository by the authors. A query for "MeMOT multi-object tracking memory" returned three repositories: mbrukman/MeMOT-pytorch (MIT) and JoshGus/MeMOT-Lite (no license) are third-party reimplementations and were not used; MCG-NJU/MeMOTR is a different paper (ICCV 2023) and was not used.
• A search restricted to the amazon-science organisation returned zero results.
• WebSearch was not permitted in this run, so a general web search was not done.

Nothing was cloned, so no code, code_commit or code_license fields are set.

Pitfalls

• Memory length conflict. §3.3 gives T_{max}=24 §3.3. §4.2 gives 22 for MOT16/17 and 20 for MOT20 §4.2. The appendix sets T_l=24 app.. Pick one value and state which benchmark it is for.
• Track-end rule not stated. The algorithm adds a state when a track's score is at or above \epsilon_{tck} alg. 1. It does not say when a track is terminated or how long an unmatched track stays alive. The frame-gap rule that decides this, if any, is not stated.
• Zeroing of absent objects is commented out. The padding rule "padded with 0 if the object does not appear" is in the memory section §3.3. The line that sets the state to zero in the algorithm is commented out in the source alg. 1.
• Threshold values are only in the appendix. The main text uses a generic \epsilon §3.5. The two values \epsilon_{pro}=0.7 and \epsilon_{tck}=0.6 appear only in app.. Results depend on them, and the paper does not show a sweep.
• Uniqueness labels for proposals. A proposal's uniqueness label depends on whether its matched object has been seen before §3.5. The exact rule for "seen before" (for example, any earlier frame, or any earlier track) is not stated.
• Loss notation. Eq. 2 uses L' for the per-entry terms and L in the detection loss eq. 2 eq. 3. Eq. 4 writes Q^t_{tc} where Q^t_{tck} is meant. The clip-level normalisation by \sum_t N_t and the per-frame averaging of the detection term are as written eq. 4. The paper does not state focal-loss \alpha and \gamma.
• Stride unit. "Increased with stride 4 at every 20 epochs" §4.2 does not say whether the stride is in frames or in clip length units.
• Learning-rate decay. "Decreased by 10 at the 100-th epoch" §4.2 is read here as dividing by 10. Warmup and weight decay are not stated.
• Data ratio. The text says MeMOT uses "5×" less training data than FairMOT §4.3, but the table gives 13.1× against 1.9× on MOT17, a ratio near 6.9 tab. 1. The ratio depends on how the multiples are counted; the table is the source.
• IDsw comparison text. On MOT17 the text pairs "34%" with TransTrack and "69%" with TransCenter §4.3. The table gives TransTrack 4614 and TransCenter 3663 identity switches against MeMOT's 2724, which is +69% and +34%, the reverse of that pairing tab. 1.
• Ablation MOTA claim. The text says the single cross-attention baseline loses "-0.51% and -0.34% MOTA" at T=3 and T=24 §4.5. The table gives 68.74 and 66.25 against 68.08, which is +0.66 and -1.83 tab. 6. The identity-switch deltas (+44 at T=3, +6 at T=24) do match the table if the order is reversed.
• Ablation numbers are not benchmark numbers. They come from a two-layer model trained on MOT17 and validated on MOT15 §4.5. Do not compare them with tab. 1.
• Long memory costs GPU memory in training. The paper names this as the limit on T and as a limitation §4.6 app..
• Annotation dependence. Training needs tracking-annotated video §4.6.
• Table numbers. Table numbers here follow the order of \input and captions in the LaTeX source. They have not been checked against a rendered PDF.

Lineage

• Builds on: DETR-style Transformer detectors §2 §3.2, Deformable DETR §3.2 §3.6, and the clip-based MOTR training scheme §3.6.
• Compared against: FairMOT, TransTrack, TransCenter, TrackFormer, CTracker, TubeTK, MOTR and others tab. 1.
• Superseded by: not stated in this card.