Class-Agnostic Counting

In plain words

Most counting tools are built for one kind of thing, such as cars or cells. This paper builds one counter that works for any kind of object. You show it a small picture of the thing you want to count, and it scans the whole image for places that look the same, then adds them up.

The model learns this "find the same thing" skill from video. Two frames of a tracked object are a natural pair of matching examples, so no counting labels are needed for that first stage. For a new kind of object, the model can be tuned with a few labelled images, changing only a small set of its parameters.

The paper tests the idea on cell images, car images from a drone and, in a smaller preliminary test, on crowds of people. It does well on some of these and not on others.

TL;DR

• Counting is cast as matching: an exemplar patch is compared with every position of a full image, giving a similarity map that is summed (or peak-picked) to count.
• The network is pretrained on ILSVRC video tracking data and then adapted with residual 1x1 adapters and batch-norm layers, training only a small fraction of parameters §2.4.
• Author-reported CARPK cars MAE is 7.48 with the full training set and 13.38 with three training images tab. 4.
• Results on VGG cells and HeLa cells are competitive rather than best-in-class. The crowd experiment is preliminary and is not state of the art tab. 2, tab. 3, tab. 5.
• The official Keras code is partial: it has pretraining, adaptation and a demo, but no evaluation, no crowd pipeline, and no pretrained-data preprocessing.

Problem

Existing counters are built for one class. Detection-based counters need a trained detector per object and struggle with few labels. Regression counters learn one mapping per class and handle overlap well §1. Domains such as microscopy have few labels because annotation needs expert knowledge.

The paper argues that object repetition inside an image can be treated as matching, and that video tracking data contains natural repeats usable for training §1 §2. It also argues that template matching (SSD, cross-correlation) is not enough for large rotations and intra-class variation §2.5.

Method

The model has three parts: embedding, matching and adapting §2 fig. 2.

Embedding §2.1. Two streams, not shared in weights after initialisation. Each is a ResNet-50 truncated after conv3_x §2.1 tab. 1. The exemplar (63x63x3) becomes a 1x1x512 vector. The search image (255x255x3) becomes an H/8 x W/8 x 512 map. Both are L2-normalised along the channel axis §2.1.

Matching §2.2. The vector is broadcast to the map size and concatenated with the map. A relation module, one 3x3 convolution and one 3x3 transposed convolution with stride 2, produces the similarity map §2.2 tab. 1. For a 255x255 image the output is 64x64 tab. 1.

Counting §2.2 §3.1. Overlapping objects are counted by summing the map (integral). Well-separated objects are counted by local maxima above a threshold T §2.2 §3.1. The map's training target uses a 100-scaled Gaussian at each instance, so the integral is read back by summing and dividing by the scale §2.3 demo.py.

Training the generic network §2.3. The GMN is trained on ILSVRC video tracking data (about 4500 videos, over 1M annotated frames) §2.3. Positive pairs are an exemplar and a search image of the same object; negatives use a different object §2.3. Input augmentation is horizontal flips, small rotations and zooms §2.3. The loss is a weighted MSE against the Gaussian target §2.3. The paper does not state the optimiser, the learning rate, the batch size or the number of epochs.

Adapting §2.4. Residual adapters are 1x1 convolutions placed in parallel with the 3x3 convolutions of the embedding module, so they change the filter responses at every stage §2.4 §2.5. During adaptation all weights are frozen except the adapters and batch-norm layers. This is 178K trainable parameters out of 6.0M, about 3% §2.4. The paper does not say how many labelled images adaptation uses in general; the car experiment uses three §3.4.

# Forward pass of the GMN  [§2.1] [§2.2] [tab. 1]
# Input:  exemplar (B, 63, 63, 3), image (B, 255, 255, 3)
# Output: similarity map (B, 64, 64, 1)

def gmn_forward(exemplar, image):
    # 1. Embed the exemplar with the first stream, 512 channels  [§2.1]
    patch_maps = phi_resnet50_conv3(exemplar)           # (B, 8, 8, 512)
    # 2. Pool the exemplar map to one vector, then L2-normalise  [§2.1]
    #    Paper: max-pool. Code: average-pool. See Paper vs code.
    vector = global_max_pool(patch_maps)                # (B, 512)
    vector = l2_normalise(vector, axis=-1)              # (B, 512)
    # 3. Embed the search image with the second stream  [§2.1]
    feature_map = psi_resnet50_conv3(image)             # (B, 32, 32, 512)
    feature_map = l2_normalise(feature_map, axis=-1)    # (B, 32, 32, 512)
    # 4. Broadcast the vector over all positions and concatenate  [§2.2]
    tiled = broadcast(vector, height=32, width=32)      # (B, 32, 32, 512)
    joint = concatenate(tiled, feature_map, axis=-1)    # (B, 32, 32, 1024)
    # 5. Relation module: 3x3 conv, then 3x3 transposed conv, stride 2  [§2.2]
    hidden = conv3x3(joint, channels=256)               # (B, 32, 32, 256)
    hidden = conv_transpose3x3(hidden, channels=256)    # (B, 64, 64, 256)
    # 6. Prediction: 3x3 conv to one channel  [§2.2] [tab. 1]
    similarity = conv3x3(hidden, channels=1)            # (B, 64, 64, 1)
    return similarity

# Training loss  [§2.3] [code: src/losses.py#L5-L14]
# Target: 100-scaled Gaussian at each instance (zeros for negatives)

def training_loss(similarity, target):
    # 1. Weight the centre region of the map more  [code: src/losses.py#L9-L13]
    #    Code: centre 16x16 gets weight 10; elsewhere weight is 0 or 1.
    # 2. Keep a random ~21% of non-centre positions, redrawn every batch
    #    [code: src/losses.py#L6-L7]
    weights = centre_weight + random_mask              # (B, 64, 64, 1)
    # 3. Weighted mean squared error  [§2.3]
    return mean(weights * (similarity - target) ** 2)

# Adapting to a new dataset  [§2.4] [code: src/model_factory.py#L122-L129]
# Input: a pretrained GMN and a few labelled images

def adapt(gmn, labelled_images):
    # 1. Freeze all weights except batch-norm, adapters and the L2 scale
    #    [code: src/model_factory.py#L122-L129] [code: src/l2_norm.py#L21-L23]
    trainable = batchnorm_and_adapter_params(gmn)
    for batch in sample_batches(labelled_images):
        # 2. Sample exemplar patches from annotated objects, augment them
        #    [code: src/utils.py#L153-L190]
        exemplars, targets = sample_exemplars(batch)
        # 3. Same weighted loss  [§2.4] [code: src/losses.py#L5-L14]
        loss = training_loss(gmn(exemplars, batch.images), targets)
        update(trainable, loss)

# Residual adapter in a bottleneck block  [§2.4]
# [code: src/resnet_blocks.py#L27-L35] [code: src/resnet_blocks.py#L75-L82]

def middle_with_adapter(x, filters):
    # 1. The 3x3 convolution and the 1x1 adapter read the same input  [§2.4]
    main = conv3x3(x, channels=filters)
    # 2. Adapter: 1x1 conv in parallel, summed with the 3x3 output  [§2.4]
    #    Code: orthogonal initialisation with gain 0.1, not in the paper.
    adapter = conv1x1(x, channels=filters)
    return batch_norm(main + adapter)

Hyperparameters

Paper values come first. Code-only values are marked and cited. The code is an official release but is not the paper's evaluation protocol.

Results

Author-reported. Setup: each dataset as listed in §3 and §4. The paper compares with methods it describes as highly engineered for those datasets §3.2. Standard deviations are over splits or exemplars as the captions state.

Synthetic VGG cells tab. 2 (N=32 split)

HeLa cells tab. 3 (MAE from detection counts)

CARPK cars tab. 4 (MAE / RMSE; several thresholds T shown)

The text reports the full-data model at T=2.75 (MAE 19.7) against the 23.8 baseline, and the best full-data MAE of 7.5 §3.4 tab. 4.

Shanghaitech Part A crowds tab. 5 (preliminary; patches chosen on validation)

Paper vs code

Official code at ab90251c1bc6fbd14cc78bf82722a2d9da729b87, from erikalu/class-agnostic-counting, owned by the first author. The README describes it as a Keras implementation of this paper README.md. The repository has no LICENSE file. It was cloned for reading only.

The search order was: the paper's links (none in the LaTeX source) and the arXiv comments field (reads only "Asian Conference on Computer Vision (ACCV), 2018"), then a GitHub repository search, which found this repository under the first author's account. No project page was linked.

• Pooling. Paper: global max-pool of the exemplar map §2.1 tab. 1. Code: global average pool model_factory.py.
• Exemplar scale. Paper: scale the search image so the exemplar's bounding box has area 63x63, i.e. s x h x w = 63^2 §2.3. Code: the search crop is pre-cut to 255 and the exemplar is a fixed 63x63 window at indices 95 to 158 of the augmented crop data_loader.py data_generator.py. The bounding-box scaling is not in this repository. The README points to external preprocessing README.md.
• Rotation, pretraining. Paper: rotations under 25 degrees §2.3. Code: angle pi/k with k drawn from 8 to 20, so 9 to 22.5 degrees utils.py. This matches the paper.
• Rotation, adaptation. Code: k drawn from 1 to 20, so angles from 9 up to 180 degrees utils.py utils.py. The paper's 25-degree limit does not apply here. Paper does not describe adaptation augmentation.
• Augmentation target. Pretraining augments only the exemplar frame; the search frame is not augmented data_generator.py. Paper says "the input data is augmented" §2.3.
• Pair sampling. Positive pairs use two frames at most 100 frames apart; negatives pair different classes data_generator.py. Paper describes positives and negatives only.
• Loss. Paper: weighted MSE, weights unstated §2.3. Code: centre 16x16 region weight 10, a random mask adds weight 1 to about 21% of positions, and the rest get weight 0 losses.py. So off-centre positions, which are most of the map, are supervised only when the mask hits them.
• Output and relation module. Paper: conv3x3 and convT3x3 with no mention of normalisation or activation §2.2 tab. 1. Code adds BatchNorm and ReLU after each of them model_factory.py and a ReLU on the final prediction model_factory.py.
• Stream initialisation. Code copies the search-stream weights into the exemplar stream at build time, then the streams train separately model_factory.py. The paper does not say this.
• Adapters during pretraining. Code keeps adapters in the network but freezes them during pretraining model_factory.py. Paper says adapters are for adaptation §2.4.
• Adaptation trainable set. Code also trains the learned L2 scale, initialised uniformly from 10 to 20 l2_norm.py. Paper lists adapters and batch norm only §2.4. The scale also trains during pretraining, since only names containing "adapt" are frozen model_factory.py.
• Adaptation input size. Code uses 800x800 inputs for cells, 360x640 for cars and 64x64 exemplars data_loader.py. The paper does not state the input size used for each dataset. The demo requires a 63x63 exemplar demo.py.
• Learning rate. Paper does not state one §2.3. Code: Adam compiled at 1e-3, then overridden each epoch by a step schedule that starts from 5e-4 model_factory.py utils.py main.py.
• Counting. Paper: sum or local maxima depending on overlap §2.2. Code: only the demo, which sums the map and divides by 100 demo.py. There is no threshold selection, no F1 computation, and no Hungarian matching in this repository. The paper's evaluation protocol §3.1 cannot be checked against code.
• Crowd experiment. The code has a crowd config data_loader.py but no crowd metadata file (the meta/ directory holds imagenet, hela_cell, vgg_cell and car) and no classification pipeline for density classes §4. The crowd result cannot be reproduced from this repository.
• Three-image car model. The paper's 3-image subset §3.4 is not documented in the README. The metadata meta/car.npz probably holds it, but it was not inspected.
• Pretrained encoder. If the ImageNet ResNet-50 weights file is missing, the code prints "could not load pretrained resnet50" and continues with random initialisation model_factory.py.

Pitfalls

• Divide the summed map by the target scale. Targets are 100-scaled Gaussians, so the count is the sum divided by 100 demo.py. The paper says "summing over the output similarity map" §2.3 and does not state the division.
• Thresholds are chosen on test in the cars table. Table 4 lists several thresholds for the same model. The paper states T is set on validation for F1 §3.1, but for cars it does not say which T the headline numbers use, and the text compares at T=2.75 §3.4 tab. 4. Treat the best-looking row with care.
• Crowd results are not state of the art. The paper's crowd method has the second-lowest MAE of five entries, behind ic-CNN tab. 5. The abstract's "competitive" wording covers cells and crowds together.
• Cells: not best MAE. On VGG cells this method has MAE 3.56, above Xie et al. (2.9), and F1 90.18, below the "full system w/o surface" (93.46) tab. 2. The paper frames it as competitive, not superior.
• Evaluation details are missing from the code. The 20-pixel tolerance for cars and the average-radius tolerance for cells §3.1 §3.4 cannot be checked against code.
• Adaptation rotations can be large. The adaptation augmentation can rotate a patch by up to 180 degrees utils.py utils.py. If you copy the pretraining settings, you get rotations up to about 22.5 degrees.
• Loss mask is random. About 21% of non-centre positions get weight 1, drawn fresh on every loss call losses.py. Runs are only comparable with a fixed random seed, which the code does not set.
• Batch norm trains on the target domain. Adaptation trains batch-norm layers, so their statistics move with the target data model_factory.py. The paper says so too §2.4. Batch size 24 is the code default main.py.
• Each exemplar is a separate forward pass. The demo runs the whole model once per exemplar and image pair demo.py. Counting five exemplars tab. 4 means five passes.
• Keras and TensorFlow versions. The README pins Python 3.8.5, Keras 2.4.3 and TensorFlow 2.2.0 README.md. The code uses tf.compat.v1 utils.py.
• Few-shot adaptation depends on frozen-weight choices. The code freezes all non-adapter, non-BN weights, and it keeps the learned L2 scale trainable. A reimplementation that freezes the scale would differ from the released code.

Lineage

• Builds on: ResNet-50 (He et al.) for both streams §2.1; residual adapters (Rebuffi et al.) for adaptation §2.4; ILSVRC video data used for tracking §2.3. None of these has a card on the wiki.
• Compared against: template matching with SSD and cross-correlation §2.5; Hsieh et al.'s regional proposal networks on CARPK tab. 4; Arteta et al. on cells tab. 2 tab. 3; Marsden et al. on adapters for counting, which uses adapters in fully connected layers §2.5.
• Superseded by: not checked against other cards on the wiki.