Focal Inverse Distance Transform Maps for Crowd Localization

In plain words

Counting people in a dense crowd is easier than finding where each one is. The usual trick is to train a network to paint a soft blob on each head and add up the paint. In a packed crowd the blobs run into each other, so the paint peaks no longer sit on heads.

This method paints a different picture. Each pixel gets a value that is high close to a head and falls off with distance, and the falloff is shaped so that the value drops to nearly zero in empty background. Every head is then a sharp peak with no overlap with its neighbours, so finding the heads means finding the peaks. A simple local-maximum check does that, and the number of peaks is the count.

The authors also add a loss that looks at small patches around each person, so the network learns the shape of each peak rather than only its height. They report that this gives better head locations than earlier methods, including on very dense and on empty scenes, and that it needs only point labels, not bounding boxes.

TL;DR

Train a regressor (HRNet-w48 in the release) to output a Focal Inverse Distance Transform map: each head is a local maximum, and the map has no overlap in dense regions §3.1 eq. 3. Train it with MSE plus an I-SSIM term over one 30x30 region per person §3.3 eq. 8. At test time, take the 3x3 local maxima above 100/255 of the peak, and call the image empty if the peak is below 0.10 §3.2.2 alg. 1. Author-reported: state-of-the-art localization on six crowd datasets and one vehicle dataset, using point labels only §5 tab. 1. The released training script does not include the I-SSIM term train_baseline.py.

Problem

Density-map regression (Gaussian blobs summed for the count) is blurry, and in dense crowds neighbouring blobs overlap, so the local maxima of the map no longer sit on heads §1 fig. 1. Detectors need box labels, which are costly in dense crowds §1 §2.1. Pseudo boxes built from nearest-neighbour distances do not match real head sizes well §1. Segmentation-style and binary maps need hand-set thresholds, and connected components can merge into one blob when heads are close §1 §2.1. The method assumes only point annotations per head §1 §4.2.

Method

Overview. A regressor predicts a FIDT map. Training uses MSE plus the I-SSIM loss. Testing finds local maxima with LMDS. Box sizes come from a k-nearest- neighbour rule, used only for visualisation §3 fig. 2.

Step 1: Euclidean distance transform. For a pixel (x, y), let P be the distance to its nearest annotated head. The authors define P over the set B of all head positions §3.1 eq. 1:

P(x,y) = \min_{(x',y') \in B} \sqrt{(x-x')^2 + (y-y')^2} \qquad [\text{eq. 1}]

Step 2: inverse distance (IDT). The raw distance has a very large range, so the authors take its inverse with a constant C that avoids division by zero §3.1 eq. 2. The paper calls this a special case of the iKNN map of Olmschenk et al. with K = 1 §3.1. Its response decays too fast near a head and stays too high in background §3.1 fig. 3.

I' = \frac{1}{P(x,y) + C} \qquad [\text{eq. 2}]

Step 3: focal IDT (FIDT). The exponent of the distance is made to grow with the distance, which slows the decay near the head and pushes background to near zero §3.1 eq. 3. The paper fixes alpha = 0.02, beta = 0.75 and C = 1 §3.1 §4 tab. 9.

I = \frac{1}{P(x,y)^{\,(\alpha P(x,y) + \beta)} + C} \qquad [\text{eq. 3}]

The FIDT ground truth is built from head points as follows §3.1 eq. 1 eq. 3:

# FIDT ground-truth map  [§3.1] [eq. 1] [eq. 3]
# Input:  image size (H, W); head points, each (x, y)
# Output: fidt map (H, W), float in [0, 1]

def fidt_map(height, width, points):
    # Distance from each pixel to its nearest head  [eq. 1]
    distance = nearest_head_distance(height, width, points)
    # Focal inverse distance with C = 1; alpha = 0.02, beta = 0.75  [eq. 3]
    exponent = 0.02 * distance + 0.75
    fidt = 1 / (distance ** exponent + 1)
    return fidt

Step 4: network. The regressor is HRNet, with one convolution and two transposed convolutions added as the representation head §3.2.1. The high-resolution branch is needed because each map peak must sit on a single head centre §3.2.1. The paper says any crowd regressor could be used instead, naming CSRNet and BL §3.2.1.

Step 5: local maxima detection (LMDS). Take a 3x3 max-pool. Keep pixels equal to their pooled value (the candidate peaks M). If the largest candidate is below T_f = 0.10, the image is treated as negative and the count is 0. Otherwise keep candidates at or above T_a = 100/255 times the largest candidate. Each kept pixel is one head, and the count is the number kept §3.2.2 alg. 1 §5.4 tab. 12.

# Local-Maxima-Detection-Strategy (LMDS)  [alg. 1] [§3.2.2]
# Input:  predicted FIDT map (H, W)
# Output: head coordinates (N, 2); count N = number of rows

def lmds(fidt_pred):
    # Keep only pixels equal to their 3x3 neighbourhood maximum  [alg. 1]
    pooled = max_pool2d(fidt_pred, kernel=3, stride=1, padding=1)
    candidates = fidt_pred * (pooled == fidt_pred)
    peak = max(candidates)
    # Negative image if the peak is below T_f = 0.10  [§3.2.2] [§5.4]
    if peak < 0.10:
        return empty_coordinates()
    # Adaptive threshold T_a = 100/255 * max(M)  [alg. 1] [§3.2.2]
    threshold = 100 / 255 * peak
    keep = candidates >= threshold
    return nonzero_coordinates(keep)

Step 6: I-SSIM loss. Standard SSIM eq. 5 is computed with lambda_1 = 0.0001 and lambda_2 = 0.0009 and turned into a loss as 1 minus SSIM eq. 6 §3.3. The authors argue that a global SSIM rewards background structure and creates false peaks §3.3. The I-SSIM loss applies SSIM to small regions, one per person, averaged over the N persons eq. 7 §3.3. The region size is 30x30 for all datasets §3.3 tab. 11. The total objective is MSE plus I-SSIM, with weight 1 on each eq. 8 §3.3.

SSIM(E,G) = \frac{(2\mu_E\mu_G + \lambda_1)(2\sigma_{EG} + \lambda_2)}{(\mu_E^2 + \mu_G^2 + \lambda_1)(\sigma_E^2 + \sigma_G^2 + \lambda_2)} \qquad [\text{eq. 5}]

L_S(E,G) = 1 - SSIM(E,G) \qquad [\text{eq. 6}]

L_{I\text{-}S} = \frac{1}{N} \sum_{n=1}^{N} L_S(E_n, G_n) \qquad [\text{eq. 7}]

L = L_{MSE} + L_{I\text{-}S} \qquad [\text{eq. 8}]

The training step, as the paper describes it §3.3 eq. 8:

# One training step: MSE plus per-person I-SSIM  [§3.3] [eq. 8]
# Input:  images (B, 3, H, W); fidt_targets (B, H, W); points per image
# Output: scalar loss

def training_step(images, fidt_targets, points):
    # Predict a FIDT map at input resolution  [§3.2.1]
    # [code: train_baseline.py#L169-L176]
    predicted = model(images)
    # Pixel-wise MSE over the whole map; reduction not stated  [eq. 8]
    mse_term = mse(predicted, fidt_targets)
    # One 30x30 region per person; centring not stated  [§3.3] [tab. 11]
    region_losses = []
    for person in persons_in(points):
        # Local crop around the person  [§3.3]
        region_pred = crop(predicted, person, size=30)
        region_gt = crop(fidt_targets, person, size=30)
        # SSIM loss on that region, lambda_1 = 1e-4, lambda_2 = 9e-4  [eq. 5]
        region_losses.append(1 - ssim(region_pred, region_gt))
    # Average over the N persons  [eq. 7]
    ssim_term = mean(region_losses)
    return mse_term + ssim_term  # weight 1 on each term  [eq. 8]

Step 7: box sizes (visualisation only). Each head's square size is s = min(f times the mean distance to its k nearest heads, 0.05 times the shorter image side), with k = 4 and f = 0.1 §4 eq. 4. The paper says the boxes do not affect localization §3.2.2 §5.1.

s_{(x,y)} = \min\left\{ f \cdot \frac{1}{k}\sum_{i=1}^{k} d^{\,k}_{(x,y)},\ \min(img\_w, img\_h) \cdot 0.05 \right\} \qquad [\text{eq. 4}]

Step 8: counting. The count is the number of LMDS points, and the result is 0 for negative images §3.2.2 §5.2 alg. 1.

Hyperparameters

Results

Author-reported. Localization is scored by precision, recall and F-measure at distance thresholds (see §4.1). Setup: NWPU-Crowd (test set from the online benchmark), JHU-Crowd++, UCF-QNRF, ShanghaiTech Part A and Part B, UCF_CC_50, TRANCOS (vehicles). Baselines are from the NWPU benchmark, from their official code, or trained by the authors where marked §5.1 §5.3.

Table 1: NWPU-Crowd localization tab. 1. Columns: validation and test sets, each at sigma_l and sigma_s (eqs. 9 and 10), giving F, P and R in percent.

* as printed in the source. Baseline rows for Crowd-SDNet and GL are reported as published; the source gives no validation figures for them tab. 1.

Table 2: UCF-QNRF localization (average over thresholds 1 to 100 pixels) tab. 2.

Table 3: JHU-Crowd++ localization tab. 3.

Table 4: ShanghaiTech Part A and Part B localization tab. 4.

Table 5: UCF_CC_50 localization tab. 5. Rows marked with a dagger were trained by the authors tab. 5.

Table 6: NWPU-Crowd counting tab. 6. The S0 to S4 columns are MAE by scene level (0; (0,100]; (100,500]; (500,5000]; >5000). Position output: the method returns coordinates.

Table 7: counting on JHU-Crowd++, UCF-QNRF, ShanghaiTech A and B, UCF_CC_50 tab. 7. Each cell is MAE / MSE.

Table 8: TRANCOS vehicle counting, GAME(L) tab. 8. Lower is better. Dagger rows were trained by the authors.

Table 9: alpha and beta on ShanghaiTech Part A, localization at sigma=8 tab. 9. The IDT row is alpha = 0, beta = 1.

Table 10: I-SSIM ablation, Part A, sigma=8 tab. 10.

Table 11: I-SSIM region size, Part A, sigma=8 tab. 11.

Table 12: threshold T_a ablation tab. 12. The dataset is not stated in the caption.

Table 13: NWPU-Crowd test, negative (S0) and extremely dense (S4) scenes tab. 13.

Table 14: other regressors with FIDT maps, NWPU-Crowd validation tab. 14. Density-map columns count by integration; FIDT columns count by localization. The FIDT runs use only the L2 loss §5.4.

Table 15: complexity tab. 15. MACs at 768 x 1024, RTX 3090 §5.5.

Qualitative results. Figures show FIDT versus density-map predictions in dense scenes fig. 6, box visualisations against LSC-CNN fig. 4 and fig. 5, and the maximum response per image used to pick T_f fig. 9.

Paper vs code

Official repository: github.com/dk-liang/FIDTM, owned by the first author (Dingkang Liang), commit e9308bdd2292e0308b2e02a9c8fc21ec5530cd9e (git rev-parse HEAD on a depth-1 clone). It is MIT-licensed. The paper's abstract gives this link. Only what is needed for the card is read; no code is quoted.

Discrepancies found:

• I-SSIM loss is not in the release. The training script uses only MSE with size_average=False train_baseline.py and sums it in the loss train_baseline.py. The paper's objective eq. 8 adds I-SSIM, which is not implemented anywhere in the repository (no ssim string in the Python or Markdown files). The reported ablation tab. 10 cannot be checked from the code.
• MSE reduction. The paper writes MSE with no reduction. The code sums over pixels and images train_baseline.py. Loss magnitudes, and so the Adam-normalised step size relative to the weight decay, depend on this choice.
• UCF-QNRF pooling in the training script. The training-time LMDS applies a 3x3 average pool to the prediction before max-pooling on UCF-QNRF train_baseline.py. The test script has that line commented out test.py. Algorithm 1 has no average pool alg. 1.
• Negative check. The code compares the global maximum of the map with T_f train_baseline.py train_baseline.py. The paper uses the maximum of the candidates M §3.2.2. The global maximum is always a candidate, so the two agree (inferred from the code).
• Adaptive threshold. The code uses 100/255 times the global maximum train_baseline.py, which matches the paper alg. 1.
• Box size. The paper's square size is min(f times the mean of k = 4 neighbour distances, 0.05 times the shorter side) eq. 4 §4. The code sums three neighbour distances and multiplies by 0.1 test.py test.py. The k-d tree query with k = 4 returns the point itself first, so the three terms are its three nearest other heads (inferred). The code then caps the value at 0.05 times the shorter side test.py, which matches. The drawn rectangle goes from the point minus sigma to the point plus sigma test.py, so its side is 2 sigma. The paper does not say whether s is a side or a half-side.
• Single-point images. The code uses a different size for an image with one head: the average image side over 4 test.py. The paper does not mention this.
• FIDT ground truth. The generator matches eq. 3 with C = 1 and alpha, beta as in the paper fidt_generate_sh.py. It also zeroes values below 1e-2 fidt_generate_sh.py, which the paper does not mention. It moves head coordinates of 0 to 1 fidt_generate_sh.py.
• Hyperparameters that match. Adam with lr 1e-4, weight decay 5e-4 and batch 16 config.py. The crop default is 256 config.py; the paper's 512 crop for the other datasets is not in any script shown.
• Regressor. HRNet-w48 with an ImageNet checkpoint config.py and a 1x1 convolution, then two transposed convolutions seg_hrnet.py. The output has no final activation seg_hrnet.py. The paper does not state the output activation.
• Training-time filter. Training images whose shorter side is under 256 are dropped train_baseline.py. The paper does not mention this.
• Training length and selection. The default is 3000 epochs config.py. Validation runs every 10 epochs from epoch 200 train_baseline.py. The best checkpoint is chosen by MAE on the test list train_baseline.py train_baseline.py. The paper does not describe selection.
• Evaluation files. The repository ships predicted coordinates and ground-truth scripts under local_eval/ [README], and the NWPU sigmas come from box annotations in the benchmark eval.py. The fixed sigma of 4 and 8 for the other datasets is in gt_generate.py.
• TRANCOS. The repository has FIDT generators for ShanghaiTech, JHU, UCF-QNRF and NWPU only [README]. The vehicle results cannot be reproduced from the release, and the paper does not say how its vehicle model was trained §5.3.

Pitfalls

• One loss term is absent from the release. The I-SSIM term, its region size and its region-centring rule are only in the paper, so a model trained with the released script is MSE-only. Its results may differ from tab. 1 and tab. 10.
• Region centring is not stated. For I-SSIM, the paper defines one 30x30 region per person §3.3 but does not say whether it is centred on the ground- truth head, on the prediction, or on a peak. A reimplementation has to choose.
• The negative threshold depends on output scale. T_f = 0.10 is set from the maxima of predicted maps on the NWPU validation set §5.4 fig. 9. A model trained with a different loss reduction or on a different scale needs its own threshold.
• Adaptive threshold ablation has no dataset. The T_a table tab. 12 gives no dataset in the caption. The text says the fixed thresholds lose to the adaptive one §5.4.
• FIDT depends on pixel distances. Eq. 1 and eq. 3 use distances in pixels, so a different resize changes the map. The paper resizes the three large datasets to a longer side below 2048 §4, and the release uses pre-resized folders make_npydata.py. The paper does not discuss how alpha and beta interact with resizing (the dependence is inferred from the formulas).
• Ties in the 3x3 maximum. Algorithm 1 keeps every pixel equal to its window maximum. A flat plateau of equal values can therefore produce several peaks for one head. The paper does not discuss ties (inferred from the algorithm).
• Label types differ across the comparison. TopoCount is trained with boxes on NWPU, while this method uses points only §5.1. The NWPU localization numbers come from the online benchmark, not from a local run §5.1.
• Inference cost. The method is slower than the real-time GL method, and the paper says it wants to move to real-time later §5.5. At 768 x 1024, it runs at 7.1 FPS on an RTX 3090 with 426.7 GMACs tab. 15.
• Small mismatches in the paper's tables. DM-Count's S0 MAE is 146.6 in tab. 6 but 146.7 in tab. 13. NoisyCC's S4 MAE is 2572.8 in tab. 6 and 2572.5 in tab. 13. The text gives the 27.4 MSE margin over SCALNet, which matches tab. 6.
• Citation key. The row labelled TopoCount in tab. 15 cites the DM-Count reference key §5.5 tab. 15. Treat the row as TopoCount, following its name and the other tables.
• Not stated. Training epochs, learning-rate schedule, ImageNet initialisation for the paper's own training, random seed, and the vehicle training protocol.
• Paper figures are described only. No figure is reproduced here. Read fig. 3 for the IDT and FIDT decay curves and fig. 9 for the peak-value distribution behind T_f.

Lineage

• Builds on: the iKNN map of Olmschenk et al. (IDT is its K = 1 case, not used for localization in that work) §3.1; the Euclidean distance transform §2.3.
• Loss ideas from: SSIM for density maps (Cao et al., DSSI-Net) §2.2 §3.3.
• Regressor: HRNet (Wang et al.) §3.2.1; the paper also tests CSRNet and BL §5.4 tab. 14.
• Compared against: TopoCount (Abousamra et al.), LSC-CNN (Sam et al.), AutoScale (Xu et al.), SCALNet, GL, Crowd-SDNet, RAZ_loc, PSDDN, VGG+GPR, TinyFaces and Faster R-CNN tab. 1. None of these have cards yet.
• Superseded by: not stated.