Part-based features

Overview

A part-based descriptor keeps some spatial or structural detail that a single global vector would average away. The image (or its patch tokens) is split into a small number of groups, each group is pooled into its own vector, and the vectors are compared between inputs 2201.09206 §3.2. The paper that defines the first variant below motivates the idea with cross-view matching: a drone and a satellite image of one building differ in position, scale and rotation, and pooled CNN features lose detail 2201.09206 §1.

The two papers form their groups differently. FSRA cuts a ranking of patches by activation 2201.09206 §3.2. Amir et al. cluster patch descriptors of a frozen model, and the parts come out of that clustering 2112.05814 §4. The FSRA paper relies on every region getting its own vector and its own loss, so the regions are trained to be discriminative, not only the global vector 2201.09206 §3.2 2201.09206 §3.4.

Variants

Activation-ordered regions (FSRA)

The regions are not fixed image crops. Each patch token of a ViT-S backbone gets a heat value, which is the mean of its feature values [arxiv:2201.09206 §3.2] 2201.09206 eq. 3. The patches are sorted from hottest to coolest, and the sorted list is cut into n consecutive regions 2201.09206 eq. 4. Each region is average-pooled into one vector 2201.09206 eq. 5. The paper uses n = 3 and reports it as the best of the values it tests [arxiv:2201.09206 §4.4]. Each region vector and the global class-token vector pass through a classifier with a cross-entropy (ID) loss, and a triplet loss is applied across views 2201.09206 §3.1 2201.09206 §3.2 2201.09206 §3.4.

On University-1652 the authors report 84.51% R@1 and 86.71% AP for drone-to-satellite with k = 3 satellite repeats, against 75.93% R@1 and 79.14% AP for LPN with ResNet-50 2201.09206 tab. 2. The k = 1 result is 82.25% R@1 and 84.82% AP 2201.09206 tab. 2. The paper's ablation shows the satellite repeats add AP on top of the triplet loss 2201.09206 tab. 7.

Card: FSRA, arXiv:2201.09206.

Clustered parts of ViT keys (Amir et al.)

No training is involved. The paper takes the keys of a pretrained DINO ViT, one of the four attention projections, as dense patch descriptors 2112.05814 §3 2112.05814 §3.1. For part co-segmentation, the foreground patches of one object are clustered into a number of parts that the user supplies, and each patch receives a part label 2112.05814 §4. The labels are then smoothed with a dense CRF. The paper names the CRF but does not state its unary terms 2112.05814 §4.

On CUB, the paper reports key-point regression errors with k = 4 parts. Its errors are 17.1, 14.7 and 19.6 (CUB-01, CUB-02, CUB-03; lower is better), which are below all three unsupervised rows of the table on each measure 2112.05814 tab. 1. The table lists a supervised foreground-mask method (Choudhury et al.) with 11.3, 15.0 and 10.6. The paper says its method is competitive with that row 2112.05814 §5.1 2112.05814 tab. 1. The same clustering step, run over the patches of several images together, gives co-segmentation 2112.05814 §4. On that task, the paper's ablation ranks keys above tokens, queries and values 2112.05814 tab. 3.

Card: Deep ViT Features as Dense Visual Descriptors, arXiv:2112.05814.

Fixed or learned parts

Not covered yet. The FSRA paper cites LPN and related part methods as the earlier part-based work 2201.09206 §2.1. No card exists for them, so this page cannot describe their mechanism.

When to use which

There are two variants with cards. They report on different tasks and benchmarks, so their numbers cannot be ranked against each other 2201.09206 tab. 2 2112.05814 tab. 1. The rows below rest only on what each paper reports.

Implementation sketch

The sketch follows the paper's equations. The region count, the backbone and the heads are the parts a variant changes.

# Region descriptors from one view  [arxiv:2201.09206 §3.2]
# Input:  patch tokens (N, D) from a ViT encoder; n regions
# Output: n region vectors (D,) and one global vector (D,)

def region_descriptors(tokens, n):
    # Class token is the global vector; the rest are patch tokens  [§3.1]
    global_vec = tokens[0]
    patches = tokens[1:]                       # (N, D)

    # Heat of a patch is the mean of its feature values  [eq. 3]
    heat = mean(patches, axis=1)               # (N,)

    # Sort patches from hottest to coolest  [§3.2]
    ordered = patches[argsort_descending(heat)]

    # Cut the sorted list into n consecutive regions  [eq. 4]
    # The last region takes the remainder
    regions = split_consecutive(ordered, n)

    # Average-pool each region into one vector  [eq. 5]
    region_vecs = [mean(r, axis=0) for r in regions]
    return region_vecs, global_vec

# Training loss over the global and region vectors  [§3.2] [§3.4]
# Each branch (global and each region) has its own classifier and triplet term

def loss(branches_a, branches_b, labels):
    # ID loss: cross-entropy per branch, then summed over both views  [§3.2]
    id_term = sum(cross_entropy(head(v), labels) for v in branches_a + branches_b)

    # Triplet loss across views only, margin 0.3  [§3.4]
    tri_term = sum(triplet(a, b, labels, margin=0.3)
                   for a, b in zip(branches_a, branches_b))
    return id_term + tri_term

# Part labels for the foreground patches of one object  [arxiv:2112.05814 §4]
# Input:  keys (n, D) from one ViT block; foreground mask (n,); k parts
# Output: a part label per foreground patch

def part_labels(keys, fg, k):
    # Keep only the foreground keys, L2-normalised  [§4]
    x = normalise(keys[fg])                    # (m, D)

    # Cluster the foreground keys into k parts  [§4]
    # k-means settings are not stated in the paper
    _, labels = kmeans(x, k)

    # Per-patch part scores are upsampled to pixels and smoothed with a
    # dense CRF; the unary terms are not stated in the paper  [§4]
    return labels

Where variants differ: how patches are scored (here, mean activation for FSRA and no scoring for the clustering paper), how they are grouped (equal-size consecutive cuts for FSRA, k-means for the clustering paper), and whether heads are shared across regions. FSRA does not say whether the region heads are shared 2201.09206 §3.2. Neither paper says how region vectors are combined at test time for the FSRA case 2201.09206 §4.2.

Contradictions

• Table 2 and Table 7 for k = 1. The k = 1 row of Table 2 (84.82 AP drone-to-satellite) matches a row of Table 7 that includes the KL term, but the text says the KL term was not used in the final model. The triplet-only row of Table 7 gives 84.85 AP 2201.09206 tab. 2 2201.09206 tab. 7 2201.09206 §3.4. The paper does not say which row Table 2 reports.
• Interpretation of the regions. The authors say the regions correspond to buildings, roads and trees 2201.09206 §4.4 2201.09206 fig. 4. The paper reports no segmentation metric for this, so the claim rests on the figure alone.
• Across the two papers. None found. They report on different tasks and benchmarks, so no result of one contradicts the other 2201.09206 tab. 2 2112.05814 tab. 1.

Papers

• FSRA, arXiv:2201.09206: activation-ordered patch regions, one vector and one loss per region, for drone-to-satellite matching.
• Deep ViT Features as Dense Visual Descriptors, arXiv:2112.05814: clusters frozen DINO ViT keys into parts with no training, with a dense CRF for per-pixel labels.