---
kind: concept
slug: part-based-features
title: Part-based features
summary: >-
  Splitting per-patch features into groups, pooling each group into its own vector and comparing those vectors. Two papers so far: FSRA orders patches by mean activation and cuts the order into 3 regions (drone-to-satellite); Amir et al. cluster frozen DINO ViT keys into parts with no training. Author-reported results on different benchmarks.
tags: [part-based, metric-learning, clustering, cross-view, geo-localization, vision-transformer, self-supervised-learning]
papers: ["2201.09206", "2112.05814"]
related: [vision-transformer, metric-learning, data-augmentation, self-supervised-learning, k-means-clustering]
status: draft
updated_at: 2026-10-18
---

## Overview

A part-based descriptor keeps some spatial or structural detail that a single
global vector would average away. The image (or its patch tokens) is split into
a small number of groups, each group is pooled into its own vector, and the
vectors are compared between inputs [arxiv:2201.09206 §3.2]. The paper that
defines the first variant below motivates the idea with cross-view matching: a
drone and a satellite image of one building differ in position, scale and
rotation, and pooled CNN features lose detail [arxiv:2201.09206 §1].

The two papers form their groups differently. FSRA cuts a ranking of patches
by activation [arxiv:2201.09206 §3.2]. Amir et al. cluster patch descriptors of a frozen
model, and the parts come out of that clustering [arxiv:2112.05814 §4]. The
FSRA paper relies on every region getting its own vector and its own loss, so
the regions are trained to be discriminative, not only the global vector
[arxiv:2201.09206 §3.2] [arxiv:2201.09206 §3.4].

## Variants

### Activation-ordered regions (FSRA)

The regions are not fixed image crops. Each patch token of a ViT-S backbone
gets a heat value, which is the mean of its feature values [arxiv:2201.09206
§3.2] [arxiv:2201.09206 eq. 3]. The patches are sorted from hottest to coolest,
and the sorted list is cut into n consecutive regions [arxiv:2201.09206 eq. 4].
Each region is average-pooled into one vector [arxiv:2201.09206 eq. 5]. The
paper uses n = 3 and reports it as the best of the values it tests [arxiv:2201.09206
§4.4]. Each region vector and the global class-token vector pass through a
classifier with a cross-entropy (ID) loss, and a triplet loss is applied across
views [arxiv:2201.09206 §3.1] [arxiv:2201.09206 §3.2] [arxiv:2201.09206 §3.4].

On University-1652 the authors report 84.51% R@1 and 86.71% AP for
drone-to-satellite with k = 3 satellite repeats, against 75.93% R@1 and 79.14%
AP for LPN with ResNet-50 [arxiv:2201.09206 tab. 2]. The k = 1 result is
82.25% R@1 and 84.82% AP [arxiv:2201.09206 tab. 2]. The paper's ablation
shows the satellite repeats add AP on top of the triplet loss [arxiv:2201.09206 tab. 7].

Card: [FSRA, arXiv:2201.09206](/papers/2201.09206).

### Clustered parts of ViT keys (Amir et al.)

No training is involved. The paper takes the keys of a pretrained DINO ViT,
one of the four attention projections, as dense patch descriptors
[arxiv:2112.05814 §3] [arxiv:2112.05814 §3.1]. For part co-segmentation, the
foreground patches of one object are clustered into a number of parts that the
user supplies, and each patch receives a part label [arxiv:2112.05814 §4]. The
labels are then smoothed with a dense CRF. The paper names the CRF but does not
state its unary terms [arxiv:2112.05814 §4].

On CUB, the paper reports key-point regression errors with k = 4 parts. Its
errors are 17.1, 14.7 and 19.6 (CUB-01, CUB-02, CUB-03; lower is better), which
are below all three unsupervised rows of the table on each measure
[arxiv:2112.05814 tab. 1]. The table lists a supervised foreground-mask method
(Choudhury et al.) with 11.3, 15.0 and 10.6. The paper says its method is
competitive with that row [arxiv:2112.05814 §5.1] [arxiv:2112.05814 tab. 1].
The same clustering step, run over the patches of several images together,
gives co-segmentation [arxiv:2112.05814 §4]. On that task, the paper's ablation
ranks keys above tokens, queries and values [arxiv:2112.05814 tab. 3].

Card: [Deep ViT Features as Dense Visual Descriptors, arXiv:2112.05814](/papers/2112.05814).

### Fixed or learned parts

Not covered yet. The FSRA paper cites LPN and related part methods as the
earlier part-based work [arxiv:2201.09206 §2.1]. No card exists for them, so
this page cannot describe their mechanism.

## When to use which

There are two variants with cards. They report on different tasks and
benchmarks, so their numbers cannot be ranked against each other
[arxiv:2201.09206 tab. 2] [arxiv:2112.05814 tab. 1]. The rows below rest only
on what each paper reports.

| Situation | Use | Why |
|---|---|---|
| Matching images of the same scene from very different viewpoints, one image per class | Activation-ordered regions | The paper's setting; it reports gains over LPN on University-1652 [arxiv:2201.09206 tab. 2] |
| No training budget, and a pretrained DINO ViT is available | Clustered parts | The method uses clustering and matching on fixed features, with no training [arxiv:2112.05814 §4] |
| You need a per-pixel part map rather than a descriptor | Clustered parts | The output is a part label per pixel, after the CRF step [arxiv:2112.05814 §4] |
| Need the region grouping to be learned | Not covered | FSRA's grouping follows activation order and reports no segmentation metric for the regions [arxiv:2201.09206 §4.4]; the clustering paper does not learn its grouping |
| Training data has one image per class on one side | Repeat that side during training | The paper's k = 3 repeats gave the best AP at batch size 8 [arxiv:2201.09206 §3.3] [arxiv:2201.09206 tab. 7] |
| Inference latency is tight | Check the backbone first | ViT-S/16 takes 1.21× the inference time of ResNet-50 in the paper's measurement [arxiv:2201.09206 tab. 3] |
| Batch size is 16 | Retune the repeat count | The paper reports k ≈ 5 is best at batch size 16 and k = 3 at batch size 8 [arxiv:2201.09206 §4.4] |
| Input images are shifted or padded at test time | Expect a drop | Black-pad and flip-pad tests lower AP for both FSRA and LPN; FSRA loses less AP than LPN at 20 px and above [arxiv:2201.09206 tab. 4] |

## Implementation sketch

The sketch follows the paper's equations. The region count, the backbone and
the heads are the parts a variant changes.

```pseudocode
# Region descriptors from one view  [arxiv:2201.09206 §3.2]
# Input:  patch tokens (N, D) from a ViT encoder; n regions
# Output: n region vectors (D,) and one global vector (D,)

def region_descriptors(tokens, n):
    # Class token is the global vector; the rest are patch tokens  [§3.1]
    global_vec = tokens[0]
    patches = tokens[1:]                       # (N, D)

    # Heat of a patch is the mean of its feature values  [eq. 3]
    heat = mean(patches, axis=1)               # (N,)

    # Sort patches from hottest to coolest  [§3.2]
    ordered = patches[argsort_descending(heat)]

    # Cut the sorted list into n consecutive regions  [eq. 4]
    # The last region takes the remainder
    regions = split_consecutive(ordered, n)

    # Average-pool each region into one vector  [eq. 5]
    region_vecs = [mean(r, axis=0) for r in regions]
    return region_vecs, global_vec
```

```pseudocode
# Training loss over the global and region vectors  [§3.2] [§3.4]
# Each branch (global and each region) has its own classifier and triplet term

def loss(branches_a, branches_b, labels):
    # ID loss: cross-entropy per branch, then summed over both views  [§3.2]
    id_term = sum(cross_entropy(head(v), labels) for v in branches_a + branches_b)

    # Triplet loss across views only, margin 0.3  [§3.4]
    tri_term = sum(triplet(a, b, labels, margin=0.3)
                   for a, b in zip(branches_a, branches_b))
    return id_term + tri_term
```

```pseudocode
# Part labels for the foreground patches of one object  [arxiv:2112.05814 §4]
# Input:  keys (n, D) from one ViT block; foreground mask (n,); k parts
# Output: a part label per foreground patch

def part_labels(keys, fg, k):
    # Keep only the foreground keys, L2-normalised  [§4]
    x = normalise(keys[fg])                    # (m, D)

    # Cluster the foreground keys into k parts  [§4]
    # k-means settings are not stated in the paper
    _, labels = kmeans(x, k)

    # Per-patch part scores are upsampled to pixels and smoothed with a
    # dense CRF; the unary terms are not stated in the paper  [§4]
    return labels
```

Where variants differ: how patches are scored (here, mean activation for FSRA
and no scoring for the clustering paper), how they are grouped (equal-size
consecutive cuts for FSRA, k-means for the clustering paper), and whether
heads are shared across regions. FSRA does not say whether the region heads
are shared [arxiv:2201.09206 §3.2]. Neither paper says how region vectors are
combined at test time for the FSRA case [arxiv:2201.09206 §4.2].

## Contradictions

- **Table 2 and Table 7 for k = 1.** The k = 1 row of Table 2 (84.82 AP
  drone-to-satellite) matches a row of Table 7 that includes the KL term, but
  the text says the KL term was not used in the final model. The triplet-only row
  of Table 7 gives 84.85 AP [arxiv:2201.09206 tab. 2] [arxiv:2201.09206 tab. 7]
  [arxiv:2201.09206 §3.4]. The paper does not say which row Table 2 reports.
- **Interpretation of the regions.** The authors say the regions correspond to
  buildings, roads and trees [arxiv:2201.09206 §4.4] [arxiv:2201.09206 fig. 4].
  The paper reports no segmentation metric for this, so the claim rests on the
  figure alone.
- **Across the two papers.** None found. They report on different tasks and
  benchmarks, so no result of one contradicts the other [arxiv:2201.09206 tab. 2]
  [arxiv:2112.05814 tab. 1].

## Papers

- [FSRA, arXiv:2201.09206](/papers/2201.09206): activation-ordered patch regions, one vector and one loss per region, for drone-to-satellite matching.
- [Deep ViT Features as Dense Visual Descriptors, arXiv:2112.05814](/papers/2112.05814): clusters frozen DINO ViT keys into parts with no training, with a dense CRF for per-pixel labels.