---
kind: concept
slug: prototype-learning
title: Prototype learning
summary: >-
  Usually one prototype per class, the mean embedding of its examples; inputs go to the nearest. FedProto shares prototypes, not weights, across federated clients (4 or 41 thousand parameters per round, versus 235 thousand or more). FedProc pulls features toward them. SDC corrects them as the embedding drifts. ProtoSeg clusters several per class for segmentation. Gains are author-reported.
tags: [prototype, federated-learning, non-iid, communication-efficiency, model-heterogeneity, contrastive-learning, class-incremental-learning, segmentation, interpretability]
papers: ["2004.00440", "2105.00243", "2109.12273", "2103.11251", "2203.15102", "2104.05919"]
related: [federated-learning, knowledge-distillation, contrastive-learning, continual-learning, semantic-segmentation, metric-learning, optimal-transport, interpretable-machine-learning, explainable-ai, zero-shot-learning]
status: draft
updated_at: 2026-11-01
---

## Overview

A prototype is a vector that inputs are compared against. In the federated and
class-incremental papers it is one vector per class: the mean of the embeddings
of that class's training examples [arxiv:2105.00243 eq. 3] [arxiv:2109.12273 eq. 5]
[arxiv:2004.00440 §3.1]. The segmentation paper keeps several prototypes per
class, each the centre of a cluster of training pixels, and it uses one per
class as its class-mean baseline [arxiv:2203.15102 §3] [arxiv:2203.15102 tab. 5].
The survey's use differs again. It describes deep prototype networks that
compare image patches with learned prototypes [arxiv:2103.11251 §4], and it
gives no class-mean definition. What the methods differ on is who shares the
prototypes, how many there are per class, and what they are used for.

Three of the four method papers label a query by the prototype it lies nearest
to: FedProto under L2 distance [arxiv:2105.00243 eq. 4], SDC as a
nearest-class-mean classifier [arxiv:2004.00440 §3.1 eq. 2], and ProtoSeg by
cosine similarity, taking each class's best prototype score [arxiv:2203.15102
§3 eq. 5–6]. FedProc uses prototypes only inside its training loss and predicts
with its own linear output layer [arxiv:2109.12273 §3.3].

In the federated case, the invariant is that the embedding layers of each
client map inputs into one common space. Every client's prototypes live in that
space, so they can be compared and averaged even when the networks behind
them differ [arxiv:2105.00243 app. B].

## Variants

### Shared per-class prototypes (FedProto)

Each client trains its own model, computes one mean embedding per class it
holds, and sends only those vectors to a server. The server averages them per
class. Each client then adds a penalty that pulls its embeddings toward the
global prototype of their label [arxiv:2105.00243 eq. 7].
Clients may hold different label subsets and different data sizes
[arxiv:2105.00243 §Experiments]. Card: [Federated Prototype Learning across
Heterogeneous Clients](/papers/2105.00243).

Reported traffic is 4×10³ parameters per round on MNIST and FEMNIST and
41×10³ on CIFAR10, against 430×10³ and 235×10³ for FedAvg [arxiv:2105.00243
tab. 1]. The exact counting rule behind these figures is not stated
[arxiv:2105.00243 tab. 1].

### Heterogeneous client architectures (FedProto-mh)

The same method where clients use different networks. The embedding size must
still match across clients [arxiv:2105.00243 app. B]. In the paper's
experiments the MNIST and FEMNIST clients differ in the channel count of one
convolution layer, and the CIFAR10 clients differ in convolution strides
[arxiv:2105.00243 §Experiments]. Accuracy is close to the single-architecture
variant on every dataset [arxiv:2105.00243 tab. 1].

### Prototype contrastive loss in federated training (FedProc)

Each client trains on a loss that mixes a prototype contrastive term with
cross-entropy. The weight on the contrastive term is α = 1 − t/T in round t of
T [arxiv:2109.12273 eq. 2] [arxiv:2109.12273 alg. 1 l.21]. The contrastive term is a softmax
over cosine similarities to the class prototypes, with the sample's own class
as the positive [arxiv:2109.12273 eq. 4]. The server averages prototypes
uniformly over clients and averages models weighted by dataset size
[arxiv:2109.12273 alg. 1 l.9-10]. As the algorithm is written, every client therefore runs the
same architecture. Author-reported top-1 accuracy with 10 clients on CIFAR-10,
CIFAR-100 and Tiny-ImageNet is 70.7%, 74.6% and 35.4%, against 69.1%, 67.5% and
27.5% for MOON [arxiv:2109.12273 tab. 2]. Card: [FedProc: Prototypical
Contrastive Federated Learning on Non-IID data](/papers/2109.12273).

Two points are left open. The paper does not say whether the positives and
negatives come from the server average of the previous round or from the
client's own class means [arxiv:2109.12273 §3.3] [arxiv:2109.12273 alg. 1 l.18]. Each round also
adds a forward pass over the full local dataset to compute prototypes
[arxiv:2109.12273 alg. 1 l.26-30]. On CIFAR-10 a round takes 11m57s, against 8m34s for FedAvg
[arxiv:2109.12273 tab. 5].

### Nearest class mean with drift compensation (SDC)

In class-incremental learning, the embedding network is trained with a triplet
loss and classified by nearest class mean. Prototypes are stored after each
task, and no old images are kept [arxiv:2004.00440 §3.1] [arxiv:2004.00440 §3.2]. The embedding
moves with each new task, so stored prototypes go stale. SDC estimates how
much each old prototype has drifted. It takes a weighted average of how the
current task's images moved, with Gaussian weights that fall off with distance
to the old prototype. The bandwidth σ is 0.3, or 0.2 on CIFAR100
[arxiv:2004.00440 §4.1] [arxiv:2004.00440 eq. 10] [arxiv:2004.00440 eq. 11]
[arxiv:2004.00440 §5]. SDC is applied on top of E-FT, E-LwF, E-EWC or E-MAS
[arxiv:2004.00440 §4.2]. Card: [Semantic Drift Compensation for
Class-Incremental Learning](/papers/2004.00440).

Author-reported gains on six-task CUB-200-2011, final task: E-FT rises from
37.4 to 49.3 and E-EWC from 54.2 to 56.4 [arxiv:2004.00440 tab. 1]. On the
eleven-task CIFAR100 and ImageNet-Subset runs, E-LwF+SDC equals E-LwF on CIFAR100
(46.1 each) and is below it on ImageNet-Subset (50.5 against 51.5)
[arxiv:2004.00440 tab. 5].

The official code applies SDC only in the evaluation script, which replays
saved per-task models. Training never reads the prototypes
[code: test.py#L235-L244] [code: train.py#L118]. The paper does not say
explicitly whether the compensation feeds back into training [arxiv:2004.00440 §4.2].

### Non-learnable cluster prototypes for segmentation (ProtoSeg)

Each class has K prototypes, each the centre of one sub-cluster of that class's
training pixels in embedding space. The sub-clusters come from online Sinkhorn
clustering, and each prototype is updated by momentum. Prototypes receive no
gradient [arxiv:2203.15102 §3]. A pixel is scored against all prototypes by
cosine similarity, and the class score is the best score among that class's
prototypes [arxiv:2203.15102 §3 eq. 5–6]. The paper's target is the learned
class weights or query vectors of softmax and query-based segmentation heads,
which it reads as parametric prototypes [arxiv:2203.15102 §2]. K = 10 is the
default; K = 20 is best in the ablation, at 46.5 against 46.4
[arxiv:2203.15102 §5.1] [arxiv:2203.15102 tab. 5]. Card: [Rethinking Semantic
Segmentation: A Prototype View](/papers/2203.15102).

Author-reported gains on ADE20K validation are +0.6 to +1.2 mIoU over four
backbones. The baselines are the authors' own reproductions with the
MMSegmentation recipe [arxiv:2203.15102 tab. 1] [arxiv:2203.15102 §5.1]. On
ADE20K with 847 classes, K = 10 scores 19.4 mIoU with 27.37 M parameters,
against 17.7 mIoU and 33.96 M for the parametric head [arxiv:2203.15102 tab. 4].
In the ablation on ADE20K with MiT-B2 at 40k iterations, cross-entropy alone
scores 45.0 and both prototype loss terms together score 46.4
[arxiv:2203.15102 tab. 5].

The released repository has one configuration, HRNet-W48 on Cityscapes. It has
no ADE20K or COCO-Stuff prototype setup [code: lib/models/nets/hrnet.py#L54-L171].

### Prototype baseline in zero-shot trigger detection (gen-arg)

This is a different use. The event-extraction paper's zero-shot trigger model
is a TapNet-style projection: each token is scored against class vectors built
from a few seed keywords, under a linear-chain CRF [arxiv:2104.05919 §2.2 eq.
5–7]. The paper does not present that model as prototype learning. It reports
plain prototype networks (Snell et al.) as a separate baseline. On ACE with no
event types seen in training, the plain prototype baseline scores 3.19 trigger
F1 (TI) and 2.22 (TC), against 55.19 and 52.10 for the TapKey model
[arxiv:2104.05919 tab. 8]. With ten frequent types seen, prototype scores and
argument heads are stronger but still below the TapKey variant
[arxiv:2104.05919 tab. 8]. Card: [Document-Level Event Argument Extraction by
Conditional Generation](/papers/2104.05919).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Clients must keep different architectures | FedProto-mh, with one shared embedding size | Prototypes are the only shared object; the embedding size is the sole constraint [arxiv:2105.00243 app. B] |
| Bandwidth or per-round upload is the bottleneck | FedProto | Tab. 1 lists 4×10³ to 41×10³ parameters per round, against 235×10³ to 430×10³ for weight averaging [arxiv:2105.00243 tab. 1] |
| Clients share one architecture and you need one global model any client can run | FedProc | It averages models and reports the accuracy of a global model [arxiv:2109.12273 alg. 1 l.10] [arxiv:2109.12273 tab. 2]. Each round costs an extra forward pass over local data [arxiv:2109.12273 alg. 1 l.26-30] |
| Non-IID image classification with one architecture and spare compute per round | FedProc | Author-reported gains over MOON of 1.6 to 7.9 points on three image datasets [arxiv:2109.12273 tab. 2] |
| You need a global classifier and no shared architecture | Not FedProto | FedProto keeps no global classifier; each client keeps its own model [arxiv:2105.00243 §Discussion] |
| Prototypes must drive prediction at inference | FedProto, SDC or ProtoSeg | FedProc predicts with its linear output layer, so prototypes only shape its training [arxiv:2109.12273 §3.3] |
| Dense prediction where a learned head grows with the number of classes | ProtoSeg | Prototype count is K per class, not a learned vector per class. At 847 classes it scores above the parametric head with fewer parameters [arxiv:2203.15102 tab. 4] |
| Segmentation, and you need a released training setup for your backbone | Check first | The repository releases only the HRNet-W48 Cityscapes configuration [code: lib/models/nets/hrnet.py#L54-L171] |
| Class-incremental learning, old images cannot be stored, embedding network | E-EWC+SDC | On CUB-200-2011 and Flowers-102 it has the highest final-task score in the paper's table [arxiv:2004.00440 tab. 1]. On CIFAR100 and ImageNet-Subset, SDC added no gain to E-LwF [arxiv:2004.00440 tab. 5] |
| No public dataset is available for distillation | FedProto | It needs none, which the paper contrasts with knowledge-distillation approaches [arxiv:2105.00243 §Introduction] |
| Each client holds very few examples per class | Check first | Accuracy falls for both FedProto and FedAvg as examples per class shrink, FedProto more slowly [arxiv:2105.00243 fig. 3] |
| Accuracy must be stable across clients | FedProto, with care | Lower spread across clients on MNIST and FEMNIST in tab. 1, but the spread figures depend on how they were computed (see Contradictions) |
| Harder label heterogeneity on CIFAR10 (appendix stdev 2) | Benchmark FedPer too | At n=3 FedPer scores above FedProto in the paper's own table [arxiv:2105.00243 tab. 4] |
| Zero-shot trigger detection with no labelled event types | Not a plain prototype network | In the paper's ACE table, the plain prototype baseline scores 3.19 TI F1 with zero seen types, against 55.19 for TapKey [arxiv:2104.05919 tab. 8] |
| Predictions must be explained by comparison with example cases | Case-based prototype networks, as a research direction | The survey lists open problems for this family: video, prior knowledge, and invalid prototypes [arxiv:2103.11251 §4] |

For FedProto, the regularizer weight λ matters. The paper sweeps it on FEMNIST
and finds the best value at 1, with accuracy dropping by about 6 points at
larger values [arxiv:2105.00243 fig. 4]. The paper's values are MNIST 1, FEMNIST
1 and CIFAR10 0.1 [arxiv:2105.00243 app. A]. The λ sweep is one dataset and the
paper does not describe how the other λ values were picked.

## Implementation sketch

Core loop of FedProto, in our own wording. Each step names its source. The
aggregation rule is the one place where the paper's equation and its code
differ, so it is marked. The FedProc, SDC and ProtoSeg cards carry their own
pseudocode [arxiv:2109.12273 alg. 1] [arxiv:2004.00440 §4.1]
[arxiv:2203.15102 alg. 1].

```pseudocode
# Server: keep one global prototype per class, starting empty
# [arxiv:2105.00243 alg. 1] [code: lib/update.py#L162-L163]
global_protos = {}

# Client: one mean embedding per class it holds  [arxiv:2105.00243 eq. 3]
def local_prototypes(embeddings_by_class):
    return {j: mean(e) for j, e in embeddings_by_class.items()}

# Server: combine each class over the clients that hold it
# [arxiv:2105.00243 eq. 6]. The paper weights each client by its share of
# the class; the released code takes a plain mean. See Contradictions.
def aggregate(client_protos):
    return {j: mean(vs) for j, vs in group_by_class(client_protos).items()}

# Client: one local step, cross-entropy plus a pull toward the global
# prototypes  [arxiv:2105.00243 eq. 7]
def local_loss(logits, embeddings, labels, global_protos, lam):
    ce = cross_entropy(logits, labels)
    # a sample whose class has no global prototype adds no pull
    # [code: lib/update.py#L162-L171]
    pull = mse(embeddings, targets_from(labels, global_protos))
    return ce + lam * pull  # lam: 1 (MNIST, FEMNIST), 0.1 (CIFAR10)
                            # [arxiv:2105.00243 app. A]

# Inference: nearest local prototype under L2  [arxiv:2105.00243 eq. 4]
def predict(x, local_protos):
    return argmin(j, l2(embed(x), local_protos[j]))
```

## Contradictions

Within the paper, the FedAvg cell for MNIST with stdev 2 and n=3 is printed
as 95.04±6.48 in the main table [arxiv:2105.00243 tab. 1] and as 91.40±6.48
in the appendix [arxiv:2105.00243 tab. 2]. Neither value is preferred here.

Eq. 6 divides by the number of contributing clients on top of count-based
weights [arxiv:2105.00243 eq. 6], which would shrink global prototypes toward
zero. The released code takes an unweighted mean instead
[code: lib/utils.py#L221-L240]. Which of the two the reported numbers come
from is not stated.

Paper and code also differ on the regularizer. The main text uses a distance
between prototypes [arxiv:2105.00243 §Methodology], the appendix uses a
squared distance between each sample's embedding and the global prototype
[arxiv:2105.00243 app. B], and the code uses mean squared error over that
same pairing [code: lib/update.py#L161-L173]. The code's running accuracy
bookkeeping may also change the spreads reported in the table
[code: lib/update.py#L456-L527] [arxiv:2105.00243 tab. 1]. These are not
disagreements between papers, and this page does not resolve them.

ProtoSeg's released numbers differ from its paper. The README reports 80.1 mIoU
for the prototype model against 79.0 for HRNet, both at 80K iterations and
batch size 8 [code: README.md#L26-L31]. The paper reports 81.1 against 80.4 on
Cityscapes validation, at 160k iterations [arxiv:2203.15102 tab. 2]
[arxiv:2203.15102 §5.1]. This page reports the paper's numbers and does not
resolve the gap.

Across papers, the numbers are not directly comparable. FedProc takes its
baseline rows from other papers and does not mark which were re-run
[arxiv:2109.12273 §4]. Its CIFAR10 setting (10 clients, Dirichlet split) also
differs from FedProto's (20 clients, n classes per client)
[arxiv:2109.12273 tab. 1] [arxiv:2105.00243 tab. 1]. ProtoSeg's segmentation
baselines are its own reproductions, and its ADE20K gains are not comparable
with FedProto's or SDC's tables.

The word "prototype" is used for different objects. The federated and
class-incremental papers use a class mean of embeddings [arxiv:2105.00243 eq. 3]
[arxiv:2004.00440 §3.1]. ProtoSeg uses K cluster centres per class, found by
clustering and not trained by gradient [arxiv:2203.15102 §3]. The survey's deep
prototype networks use learned prototypes that are compared with image patches
[arxiv:2103.11251 §4]. The event-extraction paper's baseline is a Snell-style
prototype network over class vectors for trigger detection [arxiv:2104.05919
§4.4]. Results for one kind do not transfer to the others, and this page keeps
them apart.

## Papers

- [Federated Prototype Learning across Heterogeneous Clients](/papers/2105.00243):
  per-class mean embeddings shared between federated clients, with a
  heterogeneous-architecture variant and communication counts per round.
- [FedProc: Prototypical Contrastive Federated Learning on Non-IID data](/papers/2109.12273):
  server-averaged class prototypes in a cosine-based contrastive loss for
  federated training on non-IID image data.
- [Semantic Drift Compensation for Class-Incremental Learning](/papers/2004.00440):
  nearest-class-mean classification with stored prototypes shifted by an
  estimated embedding drift, with no old images kept.
- [Rethinking Semantic Segmentation: A Prototype View](/papers/2203.15102):
  non-learnable, clustered prototypes per class replace learned class weights
  in segmentation heads, with nearest-prototype prediction.
- [Document-Level Event Argument Extraction by Conditional Generation](/papers/2104.05919):
  a prototype-network baseline for zero-shot trigger detection, compared with
  the TapNet-style trigger model the paper uses; the trigger code is not released.
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](/papers/2103.11251):
  a survey of interpretable model classes; its case-based section covers deep
  prototype networks as one open challenge area.