TransCrowd: weakly-supervised crowd counting with transformers

In plain words

Counting the people in a photo of a crowd is usually learned from pictures in which someone has put a dot on every head. Those dots are slow to add. The final answer, though, is just one number per photo. TransCrowd asks whether a model can learn to count from that number alone.

Older counters of this kind look at the picture through small windows, so each part of the model sees only a little of the scene at a time. TransCrowd cuts the picture into small squares, turns each square into a token, and lets a transformer let every token look at every other token. A final part of the network reads one number from these tokens, which is the count.

Because the model sees the whole scene at once, it can use context from across the crowd. The authors train it on counts alone and compare it with methods that used dot labels, and they note that it can match some of them.

TL;DR

TransCrowd is a crowd counter built from a pure transformer encoder. It splits each 384 x 384 tile into 16 x 16 patches, encodes them, and regresses one count per tile §3.1-§3.4. It is trained only on count labels with an L1 loss §4.1 eq. 7. The authors report that it beats the count-only methods they compare with, and that it comes close to some fully supervised ones §5 tab. 1. Every number is author-reported. The official code's regression heads differ from the paper's description, so check the Paper vs code section before implementing.

Problem

Density-map counters need a point annotation on every person to build their training targets §1. The authors argue that these point labels are expensive and are not used when the count is evaluated §1. Count-only methods so far are CNNs that regress a total count from the whole image. Their receptive field is limited, and the authors name this as the main weakness of that family §1 §2.2. The authors say these count-only methods still do not reach the results of fully supervised ones §2.2.

The authors' transformer needs pretraining. Its encoder is initialised from ImageNet weights §4.1. The paper states that CNNs outperform transformers on small datasets, and that transformers overtake them on larger ones §5. Without pretraining, the CNN baseline is better on ShanghaiTech Part A §6.5 tab. 9.

Method

Overview §3. An image is cut into fixed-size patches, and each patch is linearly projected and given a learned position embedding eq. 1 eq. 2. The sequence goes through a standard transformer encoder with no decoder §3.3. A regression head then maps the output to one number, the count §3.4. Training uses the count only, with an L1 loss over the batch eq. 7.

Two variants of the head are described §3.4.

1. TransCrowd-Token: a learned regression token is prepended to the patch sequence, and its output feeds the head. This follows the class token of BERT and ViT §3.4.
2. TransCrowd-GAP: the head takes the average over all output tokens, with no regression token §3.4 fig. 2(c).

Both heads use two linear layers. The paper does not state the hidden width or the activation §3.4. The official code does not match this description; see Paper vs code.

Encoder §3.3. The encoder has L layers. Each layer applies layer normalisation and multi-head self-attention, with a residual connection eq. 3. It then applies layer normalisation and an MLP with a residual connection eq. 4. The MLP has a GELU activation and expands the width from D to 4D, then back to D §3.3. The attention module has m independent heads whose outputs are concatenated and projected by a D x D matrix §3.3. The paper's attention equations are eq. 5 and eq. 6.

Patch and position embeddings §3.1 §3.2. The image is reshaped into N patches of K x K pixels, with N = (H/K) x (W/K) §3.1. Each patch is mapped to a D-dimensional vector by a learned matrix eq. 1. A learned position embedding is added eq. 2.

Equations (as written in the paper; the MSE in eq. 8 is a root mean square):

e = [x^1_p E;\, x^2_p E;\, \dots;\, x^N_p E], \qquad E \in \mathbb{R}^{(K^2 \cdot 3) \times D} eq. 1

Z_0 = [e_1 + p_1;\, e_2 + p_2;\, \dots;\, e_N + p_N] eq. 2

Z'_l = \mathrm{MSA}(\mathrm{LN}(Z_{l-1})) + Z_{l-1} eq. 3

Z_l = \mathrm{MLP}(\mathrm{LN}(Z'_l)) + Z'_l eq. 4

Q = Z_{l-1} W_Q, \quad K = Z_{l-1} W_K, \quad V = Z_{l-1} W_V eq. 5

\mathrm{SA}(Z_l) = \mathrm{softmax}\!\left(\frac{Q K^{\top}}{\sqrt{D}}\right) V eq. 6

L_1 = \frac{1}{M} \sum_{i=1}^{M} \left| P_i - G_i \right| eq. 7

\mathrm{MAE} = \frac{1}{N} \sum_{i=1}^{N} \left| P_i - G_i \right|, \qquad \mathrm{MSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} \left| P_i - G_i \right|^2} eq. 8

In eq. 7, M is the batch size §3.4. In eq. 8, N is the number of test images §4.3.

The forward pass, paper version. Pseudocode is in our own words; comments cite the source.

# TransCrowd forward pass  [§3.1] [§3.2] [§3.3] [§3.4]
# Input:  tiles (B, 3, 384, 384)
# Output: one predicted count per tile (B, 1)

def forward(tiles):
    # Cut each tile into non-overlapping 16 x 16 patches  [§3.1] [§4.1]
    patches = flatten_patches(tiles, size=16)        # (B, 576, 768)
    # Learned linear map from patch pixels to D channels  [eq. 1]
    embedded = patches @ projection                  # (B, 576, D)
    # Add a learned position embedding per patch  [eq. 2]
    tokens = embedded + position_embedding           # (B, 576, D)
    # Token variant only: prepend a learned regression token  [§3.4]
    if variant == "token":
        tokens = concat(regression_token, tokens)    # (B, 577, D)
    # L pre-norm encoder layers with residual connections  [§4.1] [§3.3]
    for layer in encoder_layers:
        # Self-attention sub-block  [eq. 3] [eq. 5] [eq. 6]
        tokens = tokens + attention(layer_norm(tokens))
        # MLP sub-block, width D -> 4D -> D, GELU  [eq. 4] [§3.3]
        tokens = tokens + mlp(layer_norm(tokens))
    # Token variant: read the regression token  [§3.4]
    if variant == "token":
        pooled = tokens[:, 0]                        # (B, D)
    # GAP variant: average over all output tokens  [§3.4] [fig. 2(c)]
    else:
        pooled = mean(tokens, axis=1)                # (B, D)
    # Two linear layers; hidden width not stated  [§3.4]
    hidden = linear_1(pooled)                        # (B, H), H not stated
    return linear_2(hidden)                          # (B, 1)

The heads as the official code builds them. This differs from the block above; see Paper vs code.

# Heads in the official code  [code: Networks/models.py#L21-L26]
# [code: Networks/models.py#L44-L50] [code: Networks/models.py#L63-L69]
# [code: Networks/models.py#L87] [code: Networks/models.py#L91-L98]

def token_head_as_coded(output_tokens):
    # Backbone classifier, D -> 1000; loaded from the DeiT checkpoint
    logits = backbone_head(output_tokens[:, 0])    # (B, 1000)
    # ReLU, dropout 0.5, then 1000 -> 1 linear  [code: models.py#L22-L24]
    hidden = dropout(relu(logits), p=0.5)          # (B, 1000)
    return linear_1000_to_1(hidden)                # (B, 1)

def gap_head_as_coded(output_tokens):
    # output_tokens: (B, 576, D) after the final norm; class token dropped
    # Average-pool the channel axis of each token from D to 48  [code: #L94]
    reduced = adaptive_avg_pool(output_tokens, size=48)   # (B, 576, 48)
    flat = reshape(reduced, (batch, 576 * 48))            # (B, 27648)
    # ReLU, linear 27648 -> 128, ReLU, dropout 0.5, linear 128 -> 1
    hidden = relu(linear_27648_to_128(relu(flat)))        # (B, 128)
    hidden = dropout(hidden, p=0.5)                       # (B, 128)
    return linear_128_to_1(hidden)                        # (B, 1)

Training step, count labels only. The loss below is the paper's (mean over the batch). The official code uses the sum; see Paper vs code.

# Training step with count labels only  [§4.1] [eq. 7]
# Input:  tiles (B, 3, 384, 384); one integer count per tile (B,)
# Output: scalar loss

def training_step(tiles, counts):
    # Predict one count per tile  [§3.4]
    predictions = model(tiles)                       # (B, 1)
    # Mean absolute error over the batch  [eq. 7]
    loss = mean(abs(predictions - counts))           # scalar
    # The official code sums over the batch instead  [code: train.py#L60]
    return loss

Inference for one image. The paper says six sub-images per image §4.1; the official code tiles the resized frame and sums the tile outputs.

# Count one image at test time  [§4.1] [code: dataset.py#L69-L82]
# Input:  image resized to 1152 x 768 (or 768 x 1152 if portrait)
# Output: predicted count (float)

def predict_count(image):
    # Cut the frame into six non-overlapping 384 x 384 tiles  [§4.1]
    tiles = split_into_tiles(image, size=384)        # (6, 3, 384, 384)
    # The image count is the sum of the tile outputs  [code: train.py#L219-L227]
    return sum(model(tiles))

Building training tiles and count labels. The point annotations are used only here, to derive the count of each tile.

# Build tiles and count labels  [§4.1] [code: data/predataset_sh.py#L72-L107]
# Input:  an image and its point annotations, one (x, y) per person
# Output: six 384 x 384 tiles and one integer count per tile

def make_training_tiles(image, points):
    # Resize to 1152 x 768, not keeping the aspect ratio  [§4.1]
    # [code: data/predataset_sh.py#L72-L81]
    frame = resize(image, width=1152, height=768)
    # Scale the point coordinates by the same factors  [code: #L73-L81]
    points = scale_points(points, frame_factors)
    # Three columns by two rows of tiles  [§4.1] [code: #L101-L107]
    for tile in grid(frame, size=384):
        # Count the points in the tile to get its label  [code: #L107]
        counts = count_points_inside(points, tile)
        yield tile, counts

Quick reference for the paper's own statements: the image is reshaped into K x K patches with K = 16 §4.1; N = 576 tokens per 384 x 384 tile §4.1; the encoder has 12 layers with 12 heads §4.1.

Hyperparameters

Values from the paper unless the "Where" column says code. "Not stated" means the paper does not give the value.

Results

Author-reported. Each dataset is compared with fully supervised methods (point labels) and count-only methods §5. Dataset facts are from §4.2. The paper does not describe Trancos, and the table for it reports only MAE and MSE tab. 7.

Headline comparisons, from the tables below:

• ShanghaiTech Part A tab. 1: TransCrowd-GAP reports MAE 66.1 and MSE 105.1. The count-only baselines in the same table are MATT (80.1, 129.4) and the soft-label network of Yang et al. (104.6, 145.2). MATT uses some point labels §5.
• ShanghaiTech Part B tab. 1: TransCrowd-GAP reports 9.3 and 16.1. The count-only baselines are MATT (11.7, 17.5) and Yang et al. (12.3, 21.2).
• UCF-QNRF tab. 1: TransCrowd-GAP reports 97.2 and 168.5. No count-only baseline reports UCF-QNRF in this table.
• JHU-Crowd++ test tab. 3: TransCrowd-GAP reports overall 74.9 and 295.6, against BL (75.0, 299.9), and the authors report CSRNet at 85.9 and 309.2.
• NWPU-Crowd test tab. 4: TransCrowd-GAP reports overall 117.7 and 451.0. UOT reports 83.5 and 346.9 in the same table.
• UCF_CC_50 tab. 5: TransCrowd-GAP reports 272.2 MAE, with ADCrowdNet at 257.1 MAE and MATT at 355.0.
• WorldExpo'10 tab. 6: TransCrowd-GAP averages 8.5 MAE over five scenes; LSC-CNN reports 8.0.
• Trancos tab. 7: TransCrowd-GAP reports MAE 3.23 and MSE 4.66; ADCrowdNet reports MAE 2.44.

Full tables, as printed (a dash means not reported in the paper).

Table 1 tab. 1: UCF-QNRF, ShanghaiTech Part A and Part B. "Point + count" means the method used point and count labels; "count" means count labels only.

Table 2 tab. 2: JHU-Crowd++ validation split. Low, medium and high are the count bins [0, 50], (50, 500] and above 500.

Table 3 tab. 3: JHU-Crowd++ test split, same bins.

Table 4 tab. 4: NWPU-Crowd. Validation overall, test overall, and test scene-level MAE by count range: S0 is zero, S1 is (0, 100], S2 is (100, 500], S3 is (500, 5000] and S4 is above 5000 tab. 4 caption.

Table 5 tab. 5: UCF_CC_50.

Table 6 tab. 6: WorldExpo'10, MAE per scene (S1 to S5) and the average.

Table 7 tab. 7: Trancos. Only the TransCrowd rows are count-only; the paper gives no dataset description for Trancos §4.2.

Table 8 tab. 8: run-time on a Titan XP at 384 x 384 input (FPS as printed).

Table 9 tab. 9: ShanghaiTech Part A with three initialisations. None means the model is trained directly on Part A; Pre-ImgNet fine-tunes ImageNet weights; Pre-GCC fine-tunes weights pretrained on the synthetic GCC dataset §6.5.

Table 10 tab. 10: cross-dataset evaluation, trained on the source without fine-tuning §6.6. Columns are source to target.

Paper vs code

Official code: https://github.com/dk-liang/TransCrowd at commit e4fe7ec3138f66bbeb6c3926cff6f04b03c5af58, MIT licence. The repository has no separate documentation beyond the README. The attention and block code come from the timm library and is not in this repository.

1. GAP head differs from the paper's description. The paper averages over the output tokens §3.4 fig. 2(c). The code averages over the channel axis: each of the 576 tokens is reduced from 768 to 48 values, then all 576 x 48 values are flattened into one 27,648-long vector [code: Networks/models.py#L94] models.py. The first linear layer of the head is 27,648 x 128 models.py.
2. GAP sequence keeps a class token. The GAP model still prepends a class token to the input sequence and drops its output before the head [code: Networks/models.py#L76-L77] models.py. The paper's GAP description has no such token §3.4.
3. Token head has more than the paper describes. The head is the backbone's classifier (768 to 1000), then ReLU, dropout 0.5, then a 1000 to 1 linear layer models.py models.py. The paper gives two linear layers only §3.4. The 1000-wide classifier is the timm default; the constructor does not set num_classes, and the checkpoint is loaded with strict=False models.py [inferred from timm defaults].
4. Loss is a sum, not a mean. The paper's loss averages over the batch eq. 7. The code builds nn.L1Loss(size_average=False) train.py, which sums over the batch. Adam makes updates roughly insensitive to a constant gradient scale, but the L2 weight-decay term is added to the gradient, so its relative strength drops by the batch size [inferred from the code; not checked by running it].
5. Weight decay. The paper states 1e-4 §4.1. The default in the code is 5e-4 config.py, and the README training commands do not change it README.md.
6. Grayscale augmentation is absent. The paper lists random grayscaling §4.1. The training transform has only horizontal flips dataset.py and normalisation train.py.
7. Model selection uses the evaluation split. The code evaluates on the test list every 5 epochs from epoch 10 and keeps the checkpoint with the lowest MAE train.py. The test lists are the ones used for reporting train.py. The paper does not say how the reported checkpoint was chosen.
8. Training counts come from point annotations. The preprocessing counts the annotated points inside each tile and stores that count as the label [code: data/predataset_sh.py#L86-L107]. Training then uses only the count [code: train.py#L168-L172]. Anyone reproducing the count labels from a count-only source needs the per-tile counts, which the repository does not provide.
9. Points outside the frame are dropped. A point is kept only if its integer coordinates fall inside the resized frame predataset_sh.py.
10. Aspect ratio is not kept. Each landscape image is resized to 1152 x 768 and each portrait image to 768 x 1152, with each axis scaled separately predataset_sh.py. The paper states the target size only §4.1.
11. Batch unit. The data loader's batch size counts tiles, so batch size 24 is 24 tiles train.py. The paper does not say whether it counts images or tiles §4.1.
12. Test output is not clipped. The final layer is linear with no activation models.py models.py, so a tile output can be negative, and the sum over six tiles is not clipped [code: train.py#L219-L227].
13. Scheduler. The learning rate drops by 10 at epoch 300 [code: train.py#L67]. The paper states no schedule §4.1.
14. Parameter counts fit the GAP head. Table 8 reports 90.4 M parameters for GAP and 86.8 M for Token tab. 8. The difference is about 3.6 M, which is close to the 27,648 x 128 first linear layer in the code (about 3.5 M weights) [inferred from the code's layer sizes; not counted by running it].

Pitfalls

• Two GAP heads. Implement the paper's averaged-token head or the code's channel-pooled head, not a mix. The reported GAP numbers cannot be checked against the paper text alone, and the paper does not say which head produced them. §3.4 models.py
• Equation 6 and the head width. The attention equation divides by sqrt(D) eq. 6. Each head works on width D/m §3.3. The conventional scale for each head uses the head width, not D. The paper does not say which is used. The attention code comes from timm, which is not in this repository.
• Notation in equations 5 and 6. Equation 6 writes SA(Z_l) on the left but computes Q, K and V from Z_{l-1} eq. 5 eq. 6. Read the right-hand side as the input to layer l.
• "MSE" is a root mean square. Equation 8 takes the square root, and the code does the same train.py. Tables label it MSE tab. 1. Do not compare it with an unrooted MSE from other papers.
• Text and table disagree. §6.1 says TransCrowd-GAP beats TransCrowd-Token by 2.8 MAE on Part A §6.1. Table 1 gives 69.0 and 66.1, a difference of 2.9 tab. 1. The MSE difference of 11.4 matches.
• NWPU image count. §4.2 gives 5,109 images, with splits of 3,109, 500 and 1,500 §4.2. §6.1 gives 5,190 §6.1. The splits sum to 5,109.
• Epoch count is not stated. The convergence claim is "1.9 times fewer training epochs" than CSRNet §6.3. The paper does not state the number of epochs for either method, so the claim cannot be checked.
• Run-time is one comparison. FPS is measured on one Titan XP at 384 x 384 §6.4 tab. 8. The batch size and precision are not stated.
• Results come from one run per setting. The paper reports no seeds or error bars [not stated].
• Count-only is not always pure. MATT is marked as count-level in the tables but uses some point labels §5 tab. 1. Compare with that in mind.
• Comparisons across label types. The authors call comparisons with fully supervised methods unfair §5. Their own results are close to some of them, not ahead of all of them: NWPU-Crowd tab. 4, UCF_CC_50 tab. 5, WorldExpo tab. 6 and Trancos tab. 7 each have a fully supervised method ahead.
• Tile boundaries. The six tiles do not overlap. A person whose head crosses a tile border is counted in whichever tile holds the annotated point [code: data/predataset_sh.py#L101-L107]. Each tile is predicted on its own, so a person cut by a border is counted at most once.
• Cost. Each tile gives 576 patch tokens, plus one for the token variant §4.1. Self-attention cost grows with the square of the token count §3.3. The GAP head adds a 27,648 x 128 layer, about 3.5 M weights [code: Networks/models.py#L65].
• Qualitative attention. The attention maps are shown only in one figure and are not measured fig. 4 §6.2. The paper's claim that the GAP attention is more reasonable is therefore not quantified.
• Cross-dataset results are without fine-tuning. Models are trained on the source and tested on the target directly §6.6.

Lineage

• Builds on: the Transformer encoder §3.3 §1; the ViT encoder-only design §3.3 §2.3; the class-token idea from BERT and ViT, used as the regression token §3.4; and ImageNet pretraining §4.1.
• Compared against, count-only: MATT §2.2 tab. 1; Yang et al., soft-label sorting network §2.2 tab. 1.
• Compared against, point labels: CSRNet, BL, MCNN, DM-Count, UOT, S3, ADCrowdNet and others tab. 1 to tab. 7.
• Superseded by: not stated.