An End-to-End Transformer Model for Crowd Localization

In plain words

Counting people in a crowd photo is easier than saying where each head is. This paper, CLTR, predicts the list of head positions directly. A convolutional network builds a feature map, a transformer reads it, and a fixed set of learned query slots each output one point and a confidence score. No boxes, density maps or peak-finding step are needed.

The hard part is training. Each prediction has to be paired with one real head, and in a dense crowd the simple distance rule often pairs the wrong ones. The authors add a term that looks at each head's nearby neighbours, so the pairing respects the local layout. The approach is trained end to end, and the authors report gains on several crowd datasets.

TL;DR

CLTR is a DETR-style model for crowd localization that regresses head coordinates directly, with no pseudo boxes, density maps or non-maximum suppression §1. Training uses a one-to-one Hungarian matcher whose cost adds the L1 distance between each head's mean distance to its k nearest neighbours (the KMO term) §3.3. The paper reports the best F1 on UCF-QNRF and ShanghaiTech Part A (σ=4), and a close second on NWPU-Crowd test tab. 1 tab. 2 tab. 3. All numbers are author-reported.

Problem

Crowd localization means predicting the centre of every head in an image. Datasets usually give only point labels, and box labels are expensive in dense scenes §1. Map-based methods predict density-like maps and need a non-differentiable peak-finding step and multi-scale features §1 §2.2. Detection-based methods rely on pseudo boxes and NMS §2.1. Plain DETR applied to crowds performs poorly, because its L1-based Hungarian matcher has no context and dense heads have similar confidences, which produces ambiguous matches §1 fig. 1.

Method

The model has four parts: a CNN backbone, a transformer encoder, a transformer decoder with learned queries, and two heads (points and classes). Training adds a KMO-based matcher and a loss §3 fig. 2.

1. Backbone. A CNN produces a single-scale feature map at 1/32 of the input resolution, with no feature aggregation §3 §4.1. The paper uses ResNet-50 §4.1.
2. Encoder input. A 1×1 convolution reduces the channels to c=256, the map is flattened to a sequence, and a position embedding is added §3.1.
3. Encoder. Six layers, each with multi-head self-attention (8 heads) and a feed-forward layer §3.1 §4.1 eq. 1. The attention is softmax(QKᵀ/√c)V, with the heads concatenated and projected eq. 1 §3.1.
4. Decoder. Six layers, each with self-attention, cross-attention and a feed-forward layer §3.2 §4.1. The N=500 learned queries Q_h act as the decoder's object queries §4.1. The cross-attention uses the conditional form, where the queries are the learned embeddings concatenated with the content query, following Conditional DETR §3.2.
5. Heads. Each decoded embedding gives a point (coordinates) through a regression head and a confidence through a classification head §3 fig. 2.
6. Matching (training only). Predictions are matched one-to-one with the ground-truth points by the Hungarian algorithm. The cost is the L1 distance between points plus the L1 distance between kNN-context values minus the predicted confidence §3.3 eq. 2 eq. 3. The paper says the kNN value of each predicted point is output by the network §3.3. Unmatched predictions are background §3.3.
7. Loss. Point regression is an L1 loss on the matched coordinates, normalised to [0,1] §3.4 eq. 4. Classification is a focal loss over all predictions. The total is L = L_cls + λ·L_loc with λ=2.5, and both terms are normalised by the number of instances in the batch §3.4 eq. 5.
8. Inference. Each image is split into non-overlapping patches of the training crop size, zero-padded at the border. Predictions with confidence below 0.35 are removed. The count is the number of remaining points §4.1 §4.3.

The matcher's kNN value for each ground-truth point is the mean distance to its k nearest neighbours. The paper sets k=4 §4.1 eq. 3. The ground-truth kNN value is computed from the ground-truth points themselves §3.3 eq. 3.

# KMO-based Hungarian matching  [§3.3] [eq. 2] [eq. 3]
# Input:  pred_points (N, 2), pred_conf (N,), pred_knn (N,)
# Input:  gt_points (M, 2) with M <= N, gt_knn (M,)
# Output: one prediction index per ground-truth point

def kmo_match(pred_points, pred_conf, pred_knn, gt_points, gt_knn):
    # L1 distance between coordinates, for each (gt, pred) pair  [eq. 3]
    coord_cost = l1_distance(gt_points, pred_points)     # (M, N)
    # L1 distance between kNN-context values  [eq. 3]
    knn_cost = l1_distance(gt_knn, pred_knn)             # (M, N)
    # Subtract the predicted confidence of each prediction  [eq. 2] [eq. 3]
    cost = coord_cost + knn_cost - pred_conf[None, :]    # (M, N)
    # One-to-one assignment with the Hungarian algorithm  [§3.3]
    return hungarian(cost)

# Training loss  [§3.4] [eq. 4] [eq. 5]
# Matched pairs come from kmo_match above.

def training_loss(pred_points, pred_conf, gt_points, match, labels):
    # L1 regression on the matched coordinates only  [eq. 4]
    loc = l1_loss(pred_points[match], gt_points)
    # Focal classification over all N predictions  [§3.4]
    cls = focal_loss(pred_conf, labels)
    # Both terms are divided by the number of GT instances in the batch  [§3.4]
    # Weighted sum with lambda = 2.5  [eq. 5]
    return cls + 2.5 * loc

Hyperparameters

Values marked "not stated" are absent from the paper. Code-only values are from the official repository at the commit in the frontmatter, and they are not confirmed by the paper.

Results

Author-reported. Setup: CLTR uses point labels only, a ResNet-50 at 1/32 resolution, and 500 queries §4.1. Dataset sizes: NWPU-Crowd 3109/500/1500 train/val/test §4.2; UCF-QNRF 1201/334 train/test §4.2; ShanghaiTech Part A 300/182 §4.2; JHU-Crowd++ 2272/500/1600 §4.2. Competitors that use box labels are marked with an asterisk in the paper tab. 1.

Localisation, NWPU-Crowd tab. 1

Localisation, UCF-QNRF tab. 2 (averaged over σ = 1 to 100 px)

Localisation, ShanghaiTech Part A tab. 3

Counting, NWPU-Crowd tab. 4 (position output column: whether the method outputs point positions)

Counting, UCF-QNRF, ShanghaiTech Part A and Part B tab. 5

Categorical counting, JHU-Crowd++ tab. 6 (Low: 0–50, Medium: 51–500, High: >500 people per image; Weather: degraded images)

Ablations on UCF-QNRF (the paper runs these on UCF-QNRF only, with one run per setting and no variance reported)

Transformer size tab. 7:

Matching cost tab. 8:

Number of neighbours k tab. 9:

Complexity at 1024×768 on a 3090 GPU tab. 10:

Paper vs code

Official code: dk-liang/CLTR, linked from the project page, owned by the first author, MIT licence, commit ab1b5e58eec4851cc629379abeeaceff908b48f4. I searched the project page, then cloned the repository. The paper and the code disagree in the following ways.

• KMO term is implemented differently. The paper's kNN values are a separate L1 term over the mean neighbour distance, with unit weights, and are predicted by the network §3.3 eq. 3. In the code, the mean kNN distance is appended as a third coordinate of each ground-truth point dataset.py dataset.py. The point head outputs three values conditional_detr.py, and the matcher's L1 cost runs over all three with weight 5 matcher.py matcher.py config.py.
• Class cost differs from eq. 2 and eq. 3. The paper subtracts the predicted confidence Ĉ_j eq. 2 eq. 3. The code instead uses a focal-style cost over the ground-truth class, with weight 2 matcher.py config.py.
• k is effectively 3, not 4. The paper sets k=4 and averages over k neighbours §4.1 eq. 3. The code queries 4 points, drops the first column (the point itself) and averages the rest, so it averages 3 neighbours dataset.py. The paper's sum over k is ambiguous on whether the point itself is counted.
• Regression loss covers the kNN value. The paper's regression loss is over coordinates only eq. 4. The code applies the L1 loss to all three outputs, including the kNN value conditional_detr.py.
• Loss weights. The paper uses weight 1 for classification and λ=2.5 for regression §3.4 eq. 5. The code uses 2 and 5 config.py. The ratio 5:2 equals λ=2.5, but the absolute scale differs.
• Normalisation. The paper normalises both terms by the number of instances in the batch §3.4. The code divides the regression loss by the total ground-truth count across the batch, averaged over processes conditional_detr.py conditional_detr.py. The classification term is further multiplied by the number of queries conditional_detr.py.
• Auxiliary losses. The code adds a loss at every decoder layer by default conditional_detr.py conditional_detr.py config.py. The paper does not mention them.
• Queries. The paper gives N=500 §4.1. The NWPU-Crowd script trains with 700 queries nwpu.sh. Section 5.4.1 reports that 700 queries lower UCF-QNRF performance (MAE 86.8 vs 85.8) tab. 7.
• Augmentation. The paper lists crop, scale and flip §4.1. The code adds grayscale augmentation, which the paper does not mention config.py jhu.sh. The flip probability of 0.5 matches dataset.py.
• Patch padding. The paper says zero padding §4.1. The code pads on the left and top only dataset.py.
• Crop filter. The code rejects training crops with as many or more ground-truth points than queries, or fewer than min_num (default -1) dataset.py. The paper states the observation that all patches contain fewer than 500 people §5.4.1 but not this rule.
• Normalisation docstring. The criterion's docstring says targets are normalised by image size conditional_detr.py. The data code divides by the crop width dataset.py. The paper says [0,1] §3.4.
• Backbone. The code freezes conv1, layer1 and all BatchNorm layers backbone.py backbone.py. The paper does not state any freezing.
• Single learning rate. The code has an lr_backbone argument, but the optimiser uses one rate for every parameter train_distributed.py config.py. The paper states one rate and no backbone rate.

Pitfalls

• The kNN statistic is crop-local. Ground-truth points are taken from the training crop only dataset.py, so points near the crop border have fewer neighbours than the same points would have in the full image. The paper does not say whether the kNN values are computed per crop or per image.
• The kNN value is a learned output. Matching depends on a predicted quantity that the network must learn §3.3. The paper does not show how well the kNN head is trained, and the ablation does not separate this from the coordinate head.
• Small point counts use other branches. Crops with 1 to 3 points take separate branches in the kNN code dataset.py. I read only part of them, so check these before reimplementing.
• Abstract vs table. The abstract says the method achieves the best localisation on NWPU-Crowd, but on the NWPU test set TopoCount has a higher F1 (69.1 vs 68.5) tab. 1. The text calls it "competitive" §5.1. TopoCount uses box labels in training.
• Ablations are single runs. The paper gives no seeds or spread. The gap between L_m and L_m^k (MAE 91.3 vs 85.8) is one run tab. 8. The code's default seed is 1 config.py.
• Inference cost. The model has 43.4M parameters, the most in the comparison, though it has the fewest MACs at 1024×768 tab. 10. Matching runs on the CPU with SciPy per image matcher.py, and the cost matrix is moved to CUDA matcher.py.
• Patch borders. Non-overlapping patches with zero padding can split or miss heads at patch borders. The paper names fixed-size crops as the cause of failure on very large heads §5.5 fig. 5.
• Query count caps crop density. Training keeps only crops whose point count is below the query count dataset.py, and the paper picks N=500 §4.1. Table 7 shows 700 queries slightly worse than 500 on UCF-QNRF tab. 7. The paper does not say what to do for crops above N.
• Reported metrics. NWPU-Crowd uses a per-head σ from head size §4.3. The paper does not state how the head width and height were obtained for each annotation.

Lineage

• Builds on: DETR (Carion et al., 2020) for set prediction with a transformer decoder and a Hungarian matcher §2.4 §3.3.
• Builds on: Conditional DETR (Meng et al., 2021) for the conditional cross-attention §3.2.
• Compared against: TopoCount, AutoScale, LSC-CNN, GL, RAZ_loc, CL, and box-based detectors tab. 1 tab. 2 tab. 3.
• Concurrent work: P2PNet, a regression-based crowd localiser that uses 8×W×H point proposals §2.3.
• Superseded by: not known from content in this wiki.