DecideNet: Counting Varying Density Crowds Through Attention Guided Detection and Density Estimation

In plain words

Counting a crowd from a camera image is hard because crowds are not equally dense everywhere. Some parts of a picture hold a few well-separated people, and other parts hold a dense mass. There are two common ways to count. One finds each person and counts the detections. This works well when people are far apart, but it misses many people when they overlap or are tiny. The other learns a "density" map that adds up to the count without finding anyone. This works well in dense areas, but in sparse areas it tends to over-count.

DecideNet runs both methods at once and learns when to trust which one. Each part of the image gets a score that says how much to rely on the detector and how much on the density estimate. The two maps are blended with those scores, and the total of the blended map is the crowd count. The scores are learned during training, and a detection-based signal helps teach them where the detector can be trusted.

TL;DR

DecideNet is an end-to-end crowd counter that outputs one density map from two sources: a detector's head boxes turned into Gaussian bumps, and a small regression network. A pixel-wise attention network chooses the blend. The authors report the best MAE among the compared methods on Mall and WorldExpo'10, and a competitive but not best MAE on ShanghaiTech PartB. Code: none found (see Paper vs code).

Problem

Crowd counting must cope with density that changes across space and time within one scene §1 fig. 1. The authors report that detection scores drop as the ground-truth count per patch rises, so detection under-counts in crowded areas §1 fig. 1(c). They report that regression maps over-count in sparse areas and in background regions, because they do not know where people are §1 fig. 1(d). Neither method alone is judged adequate across the whole density range §1.

Method

The model has three CNN blocks, parameterised by Ω = (Ω_det, Ω_reg, Ω_qua) and trained jointly §3:

1. RegNet produces a regression density map D^reg from the image patch, without locating heads §3.2 fig. 3. It is a five-layer fully convolutional network with a ReLU after the last layer §3.2 eq. 2.
2. DetNet is a Faster R-CNN head detector on a ResNet-101 backbone §3.3. A fixed Gaussian is placed at the centre of each detected box and the results are summed into D^det, so D^det is "location aware" §3.3 eq. 3 fig. 4.
3. QualityNet takes D^det, D^reg (both upsampled to image size) and the image I_i stacked into five channels. Four convolutional layers and a pixel-wise sigmoid output an attention map K with values in (0, 1), where a larger value means "trust detection here" §3.4 fig. 5.
4. The final density is the attention-weighted sum of the two maps §3.4 eq. 4:

D^{final}(p) = K(p) \odot D^{det}(p) + (\mathbf{J} - K(p)) \odot D^{reg}(p) eq. 4

The count of a region is the sum of D^final over its pixels §3.1.

Ground truth is a density map made by placing a Gaussian at each annotated head point and summing; the sum over pixels equals the head count §3.1 eq. 1.

The training loss is the sum of three terms, one per block §4 eq. 5:

L_{decide} = L_{reg} + L_{det} + L_{qua} eq. 5

• L_reg: pixel-wise squared error between D^reg and the ground-truth density, averaged over training images §4 eq. 6.
• L_det: the Faster R-CNN classification and box-localisation losses, with the ground-truth boxes made from the average box size §4 eq. 7.
• L_qua: the pixel-wise squared error between D^final and the ground truth, plus λ times the squared difference between K and the detection-score map S^det §4 eq. 8. The authors call this the "quality-aware" loss §4.

Per iteration, the three loss gradients are computed in turn, and each one updates only its own block's parameters §4.

# DecideNet training step  [§3] [§4]
# Input:  patch (3, h, w); head points and boxes for the patch
# Output: L_decide = L_reg + L_det + L_qua  (scalar)

def decidenet_loss(patch, head_points, boxes_gt, num_images):
    # RegNet: 5-layer fully convolutional regression map  [§3.2] [fig. 3]
    density_reg = regnet(patch)                       # (h', w'), ReLU output
    # DetNet: Faster R-CNN head boxes, Gaussian at each box centre  [§3.3]
    boxes, scores = faster_rcnn(patch)
    density_det = gaussian_at_centres(boxes)          # sigma not stated
    # Per-pixel detection score map S_det, 0.1 outside all boxes  [§5.1]
    score_map = fill_scores(boxes, scores, default=0.1)
    # Bring both maps to image size for QualityNet  [§3.4]
    density_reg_up = upsample(density_reg, patch)     # method not stated
    density_det_up = upsample(density_det, patch)
    # QualityNet: attention K in (0, 1) per pixel  [§3.4] [fig. 5]
    stacked = concat(density_det_up, density_reg_up, patch)
    attention = qualitynet(stacked)                   # (h, w), sigmoid
    # Attention-weighted fusion of the two maps  [§3.4] [eq. 4]
    density_final = (attention * density_det
                     + (1 - attention) * density_reg)
    # Regression loss: squared error summed over pixels, over N  [§4] [eq. 6]
    loss_reg = sum_pixels((density_reg - gt_density) ** 2) / num_images
    # Detection loss: Faster R-CNN cls + loc, box size = average  [§4] [eq. 7]
    loss_det = faster_rcnn_loss(boxes, boxes_gt) / num_images
    # Quality loss: final error + lam * (K - S_det)^2  [§4] [eq. 8]
    loss_qua = (sum_pixels((density_final - gt_density) ** 2)
                + lam * sum_pixels((attention - score_map) ** 2)) / num_images
    # Count of the region is the sum of density_final  [§3.1]
    return loss_reg + loss_det + loss_qua

Hyperparameters

Results

Author-reported. All numbers are MAE and MSE on each dataset's test split §5.1. The paper does not write a formula for MSE beyond "mean squared error" §5.1.

Mall dataset (320 × 240 frames) §5.2 tab. 1:

ShanghaiTech PartB §5.3 tab. 2:

DecideNet+R3 adds an extra R3 stream taken from Switching-CNN §5.3.

WorldExpo'10, MAE per scene, counting only inside the ROI §5.4 tab. 3:

Ablation on Mall and SHB §5.5 tab. 4:

The paper reports that RegNet alone beats DetNet alone on both datasets, that late fusion helps on SHB but only mediocrely on Mall, and that the attention block roughly halves the MAE of late fusion. Adding the quality-aware loss lowers error further on both §5.5. The per-image count curves are in fig. 6.

Paper vs code

No official code.

The paper names no repository, no project page and no code availability statement [arXiv abstract, §1–§6]. A GitHub repository search for "DecideNet" returned four repositories, none owned by an author: kbonna/decidenet is an fMRI and behavioural study with the same project name, and the other three are personal repositories. Further searches in this session returned HTTP 403, so the search is not exhaustive.

Pitfalls

• Density resolution. RegNet has two 2 × 2 max-pools §3.2, so its output is smaller than the patch. The paper defines D^reg per pixel of I_i §3.2 but then says QualityNet "upsamples" both maps to image size §3.4. The upsampling method is not stated, and it is not stated whether the final blend in eq. 4 uses the upsampled or native-size maps.
• Detector Gaussian. The σ of the Gaussian placed on detected centres eq. 3 is not stated. Detection thresholding and non-maximum suppression are also not stated, though both change D^det directly.
• Gradient path to the detector. D^det depends on the detected box centres, which are discrete outputs. How gradients from L_qua reach Ω_det, and whether the score map S^det is detached, is not stated.
• Loss scale. L_reg and L_qua sum over pixels and divide by N, the number of training images eq. 6 eq. 8, not by the pixel count. The batch size and whether N is per batch or per dataset are not stated. The value of λ is not stated, so the balance between the two parts of L_qua is unknown.
• Distance type. The text says "Euclidean distances" §4, but eq. 8 uses a squared norm. Taking the root changes the scale of the term.
• Alternating updates. "Alternatively calculated" §4 could mean one loss per iteration or all three each iteration. The paper does not say.
• Per-iteration detector cost. The detection score map is produced by running DetNet on every training patch in every iteration §5.1. The training cost of this is not reported, and inference cost is not reported.
• Box supervision. Box sizes are taken as the average over a hand-labelled subset of training heads §4. The subset size and whether the average is per dataset are not stated, and the authors say this constant is used for all heads. Accuracy on scenes with very different head sizes is not evaluated in the paper.
• Inference procedure. Training uses patches §5.1. Whether test images are processed whole or in patches, and how patches are stitched, is not stated.
• Random noise units. Noise of [−5, 5] per pixel §5.1 needs the pixel intensity scale, which is not stated.
• Headline claims. The abstract and conclusion claim state-of-the-art on all three datasets §6. The tables show CP-CNN with a lower SHB MAE (20.1 against 21.53) tab. 2, and Cross-scene with lower per-scene MAE on S4 and S5 tab. 3. The authors attribute the S4 gap to the detector working poorly on remote crowds §5.4.
• Missing details. Optimizer momentum, weight decay, batch size, the pretraining of the Faster R-CNN backbone, and the validation metric are not stated.

Lineage

• Builds on: Faster R-CNN [ren2015faster] for the detection branch §3.3, and Gaussian-kernel density-map ground truth, as used in the cited density regression work [onoro2016towards] §3.1.
• Compared against: detection baselines (R-FCN, Faster R-CNN, SquareChn) and regression baselines (M-CNN, Switching-CNN, CP-CNN, Cross-scene, FCN) §5.2–§5.4.
• Superseded by: not stated in this wiki.