POI: Multiple Object Tracking with High Performance Detection and Appearance Feature

In plain words

Following several people through a video means finding each person in every frame and linking the detections into one track per person. The authors argue that most tracking papers focus on the linking step and use weak detections. They show that better detections and a learned "who is this?" description of each person's appearance make a simple tracker work well.

The paper has two trackers. The online one looks only at past frames: it predicts where each track should be, compares it with the new detections, and links them. The offline one sees the whole video and links short pieces into longer tracks. The authors release their detections and appearance features, which are the main reusable part of the work.

TL;DR

• Detection and appearance are treated as the main levers: a Faster R-CNN detector and a 128-d appearance embedding trained with softmax and triplet losses §1, §2.
• Online tracker: Kalman prediction plus Kuhn-Munkres matching on an affinity that multiplies appearance, motion and shape terms §3.
• Offline tracker: an improved H2T with dense-neighbour association and scale gating for big targets §4.
• Author-reported MOT16 challenge: online MOTA 66.1, offline MOTA 68.2 tab. 4.
• Relevant if you build tracking-by-detection systems. Several parameters (learning rates, triplet margin, Kalman noise) are not stated (see Pitfalls).

Problem

In data-association tracking, the detections drive the result: a weak detector causes false negatives and false positives that the tracker cannot recover from §1. The authors also note that the association step is often the only focus of prior work, with hand-crafted appearance features [abstract]. Global assignment (Kuhn-Munkres) can fail when detections are missing, which motivates the two-stage matching in the online tracker §3. The offline tracker is slow on long sequences with many targets, because the affinity matrix is large §4, "Algorithm Efficiency".

Method

The card covers four parts: the detector §1, the appearance embedding §2, the online tracker §3 and the offline tracker §4. The paper uses "Section 2" as its own cross-reference, so the section numbers below follow the compiled order: Detection is §1 and Conclusion is §7. The paper has no Introduction section in the source.

1. Detector §1. A Faster R-CNN [RenHGS15] is fine-tuned from a VGG-16 model pretrained on ImageNet. Training data are the ETHZ and Caltech pedestrian sets and a self-collected surveillance set of 365,653 boxes in 47,556 frames §1. Multi-scale training picks a random pyramid scale for each training sample; at test time a single scale and a single model are used §1. Skip pooling [BellZBG15] and multi-region features [GidarisK15] combine features from several scales and levels §1. The paper states that these choices reduce the sum of false positives and false negatives on MOT16 train tab. 1.

2. Appearance embedding §2. The network is similar to GoogLeNet [SzegedyLJSRAEVR15], with three changes: the input is 96 x 96 pixels, the pool5 kernel is 3 x 3 instead of 7 x 7, and the output is a fully connected layer with a 128-d feature §2. Tracking crops each detection box, resizes it to 96 x 96 and extracts the feature §2. Affinity uses cosine distance §2. Training data are about 119 K patches from 19,835 identities, combining PRW, Market-1501, VIPeR and CUHK03 §2. The loss is a softmax loss, which makes the feature discriminative, plus a triplet loss, which makes same-identity cosine distances small §2.

3. Online tracker §3. Motion is predicted with a Kalman filter; data association uses the Kuhn-Munkres (Hungarian) algorithm §3. Algorithm 1 gives the per-frame procedure alg. 1.

Affinity between tracklet i and detection j is the product of three terms §3, eq. 4:

• appearance: cosine similarity of the two features §3, eq. 1;
• motion: an exponential of the squared centre offset, normalised by the detection width and height, with weight w1 §3, eq. 2;
• shape: an exponential of the relative differences in height and width, with weight w2 §3, eq. 3.

The two-stage matching first pairs high-quality tracklets with all detections, then pairs the remaining high-quality tracklets and all low-quality tracklets with the unmatched detections §3, "Data Association". A pair counts as a success when its affinity passes tau_a alg. 1, line 4. Tracklets are split into high and low quality by tau_t alg. 1, line 2. A tracklet that is lost for more than tau_m frames is dropped [§3, "Threshold Value Setting"].

Tracking quality is the mean affinity over all successful couples in the tracklet's history, multiplied by a term that grows with length §3, eq. 5.

# Online tracker, one frame  [alg. 1] [§3]
# Input:  detections (D), tracklets from the previous frame (T_prev)
# Output: tracklets for the current frame (T_new)

def track_frame(detections, tracklets_prev):
    # Affinity between every tracklet and every detection  [alg. 1, line 1]
    affinity_matrix = build_affinity(tracklets_prev, detections)
    # Split tracklets by quality threshold tau_t  [alg. 1, line 2]
    high, low = split_by_quality(tracklets_prev, threshold=tau_t)
    # Stage 1: high-quality tracklets against all detections  [§3]
    pairs = hungarian(high, detections, affinity_matrix)
    # Keep pairs whose affinity passes tau_a  [alg. 1, line 4]
    success = [p for p in pairs if p.affinity >= tau_a]
    # Stage 2: unmatched high and all low tracklets against the rest  [§3]
    rest_tracks = (high - success.tracks) + low
    rest_dets = detections - success.dets
    pairs_2 = hungarian(rest_tracks, rest_dets, affinity_matrix)
    success += [p for p in pairs_2 if p.affinity >= tau_a]
    # Kalman update and feature averaging for matches  [alg. 1, line 6]
    updated = [kalman_and_average(p.track, p.det) for p in success]
    # Predict unmatched tracklets, drop those lost for more than tau_m frames
    # [alg. 1, line 7]
    predicted = [predict_or_drop(t, max_lost=tau_m) for t in failed_tracks]
    # Unmatched detections start new tracklets  [alg. 1, line 8]
    new = [init_tracklet(d) for d in unmatched_dets]
    # Keep only tracklets inside the image  [alg. 1, lines 9-10]
    return filter_inside_image(updated + predicted + new)

# Pairwise affinity for the online tracker  [§3, eq. 1-4]
# trk and det carry centre (x, y), width (W), height (H) and feat (128-d)

def affinity(trk, det, w1, w2):
    # Appearance: cosine similarity of the features  [§3, eq. 1]
    aff_app = cosine(trk.feat, det.feat)
    # Motion: centre offset normalised by the detection size  [§3, eq. 2]
    dx = (trk.x - det.x) / det.W
    dy = (trk.y - det.y) / det.H
    aff_mot = exp(-w1 * (dx ** 2 + dy ** 2))
    # Shape: relative height and width differences  [§3, eq. 3]
    d_h = abs(trk.H - det.H) / (trk.H + det.H)
    d_w = abs(trk.W - det.W) / (trk.W + det.W)
    aff_shp = exp(-w2 * (d_h + d_w))
    # Final affinity is the product of the three terms  [§3, eq. 4]
    return aff_app * aff_mot * aff_shp

# Tracking quality of a tracklet  [§3, eq. 5]
# couples: all successful (tracklet, detection) pairs in its history

def tracking_quality(tracklet, couples, w3):
    # Mean affinity over the history of successful couples  [§3, eq. 5]
    mean_affinity = sum(c.affinity for c in couples) / len(tracklet)
    # Growth term: rises towards 1 as the tracklet gets longer  [§3, eq. 5]
    growth = 1 - exp(-w3 * sqrt(len(tracklet)))
    return mean_affinity * growth

4. Offline tracker §4. The offline tracker is an improved H2T [WenLYLYL14] that uses dense-neighbour (DN) search built on K-Dense Neighbors [LiuYLY12]. Algorithm 2 gives the procedure alg. 2. The video is cut into disjoint temporal segments; DN search links detections into short tracklets inside each segment; neighbouring segments are then merged and DN search runs again, until one segment remains alg. 2. The paper says the appearance term is the CNN feature of §2, and that it drops H2T's high-order information to shrink the affinity matrix [§4, "Appearance Representation", "Algorithm Efficiency"]. For targets of very different sizes, a detection is not linked to a target when the ratio of their scales is below tau_s, and motion and smoothness weights are reduced when the detection height is above tau_r of the image height §4, "Big Target".

# Offline tracker  [alg. 2] [§4]
# Input:  all frames of a video, with detections per frame
# Output: trajectories of targets

def track_offline(frames, detections):
    # Cut the video into disjoint temporal segments  [alg. 2, line 1]
    segments = split_in_time(frames)
    # Link detections into short tracklets inside each segment  [alg. 2, line 2]
    tracklets = [dn_search(s, detections) for s in segments]
    # Merge nearby segments and re-run DN search, until one segment remains
    # [alg. 2, lines 3-6]
    while len(segments) > 1:
        segments = merge_nearby(segments)
        tracklets = [dn_search(s, tracklets) for s in segments]
    return tracklets

Hyperparameters

Results

Author-reported. The challenge table is labelled "MOT16 Rank List" and the caption does not restate the split. The ablations use the MOT16 train set and tune only the detection score threshold on it §5.

Detection, MOT16 train set tab. 1:

Online tracker, MOT16 train set tab. 2:

Offline tracker, MOT16 train set tab. 3:

MOT16 rank list, comparison with other trackers tab. 4:

The paper reports that both trackers beat the listed state of the art "by a large margin" and that the offline tracker has the fewest false negatives, with more false positives because of the interpolation module §6.

Paper vs code

No official code. The paper's abstract links a Google Drive folder that holds the authors' detections and appearance features, not a code repository [abstract, footnote 1]. Searches run for this card: the GitHub repository search for "POI tracker sensetime", "KDNT dense neighbors tracker" and "POI multiple object tracking"; the user search for "fengwei yu"; and the repositories of the organisation detectRecog. None of them is a POI or KDNT implementation, and the org's repositories are other projects. No matching repository was cloned, so there is nothing to compare against the paper.

Pitfalls

• Equation numbers. The TeX source has no equation labels. The numbers used above (eq. 1 to eq. 5) follow the order of the display equations in Section 3. Check them against the compiled PDF.
• Affinity weights. The text says the three affinities are combined "with weights w1 and w2" §3, but the printed formula (eq. 4) multiplies the three terms with no extra weight. The weights appear only inside the motion and shape terms (eq. 2 and eq. 3). Implement eq. 4 as written, and expect this to be the first question a reader hits.
• Coordinates. The paper does not say whether X and Y are box centres or corners, nor how H and W are defined in the motion and shape terms. Centres are the usual reading, but this is not stated.
• Two-stage matching. Algorithm 1 line 3 describes one matching between all high and low tracklets and the detections alg. 1, line 3. The text describes two stages §3, "Data Association". The card follows the text.
• Success threshold. tau_a is applied "to decide whether association success or not" alg. 1, line 4 but the paper does not say whether the test is >= or >.
• Quality scale. tau_t = 0.5 is applied to the quality metric eq. 5, but the range of the metric is not stated, so the threshold cannot be checked against it.
• Kalman filter and feature averaging. The state vector, noise values and the averaging rule are not described. Algorithm 1 only says "Average".
• Training recipe. Learning rates, schedules, batch sizes, epochs, the triplet margin, the triplet sampling scheme and the relative weight of the two losses are all unstated. An implementer cannot reproduce the embedding from the paper alone.
• Per-sequence detection thresholds. The text says the threshold is tuned on the train set and applied to similar test scenes §5. But the 0.1 value for MOT16-03 and MOT16-04 is justified by the surveillance scene and by the self-collected training data, not by a reported tuning run. Treat it as a dataset-specific choice.
• Label mismatch. Tables 2 and 3 call the baseline feature "GoogLeNet feature", while the text calls it the "default CNN feature" §5. The paper does not say which network the baseline feature is, so the table label cannot be taken at face value.
• Interpolation. Section 6 credits the offline tracker's false-positive count to an "interpolation module" §6, but no section describes it. Treat the offline pipeline as incomplete for reproduction.
• Offline parameters. Segment length, merge rule and the DN-search parameters are not stated. Algorithm 2 gives only the loop structure.
• Data availability. The self-collected surveillance set is used for detector training but the paper does not state whether it is released. The Google Drive link in the abstract is described as holding "our detection and appearance feature" [abstract]; the card has not inspected it.
• Training data terms. The paper does not state the terms of use for the ETHZ, Caltech, PRW, Market-1501, VIPeR or CUHK03 datasets. Check them before reusing the trained models.
• Ablations are on the train set. Tables 2 and 3 use the MOT16 train set; only Table 4 compares with other trackers. Do not read the ablations as test-set results.

Lineage

• Builds on: Faster R-CNN for detection; skip pooling and multi-region features for detector features §1; a GoogLeNet-like network as the base of the embedding §2; Kalman filter and Kuhn-Munkres matching for tracking §3.
• Extends: H2T (Wen et al., 2014), with the appearance feature of §2 replacing its hand-crafted feature, dropped high-order information, and scale gating for big targets §4. Uses K-Dense Neighbors (Liu et al., 2012) for DN search §4.
• Compared against: KFILDAwSDP (online) and MCMOT-HDM (offline) on MOT16 tab. 4; the official DPMv5 detector on MOT16 train tab. 1; the GoogLeNet-feature variant tab. 2, tab. 3.
• Superseded by: not stated in this paper.