Prototype learning

Overview

A prototype is a vector that inputs are compared against. In the federated and class-incremental papers it is one vector per class: the mean of the embeddings of that class's training examples 2105.00243 eq. 3 2109.12273 eq. 5 2004.00440 §3.1. The segmentation paper keeps several prototypes per class, each the centre of a cluster of training pixels, and it uses one per class as its class-mean baseline 2203.15102 §3 2203.15102 tab. 5. The survey's use differs again. It describes deep prototype networks that compare image patches with learned prototypes 2103.11251 §4, and it gives no class-mean definition. What the methods differ on is who shares the prototypes, how many there are per class, and what they are used for.

Three of the four method papers label a query by the prototype it lies nearest to: FedProto under L2 distance 2105.00243 eq. 4, SDC as a nearest-class-mean classifier 2004.00440 §3.1 eq. 2, and ProtoSeg by cosine similarity, taking each class's best prototype score [arxiv:2203.15102 §3 eq. 5–6]. FedProc uses prototypes only inside its training loss and predicts with its own linear output layer 2109.12273 §3.3.

In the federated case, the invariant is that the embedding layers of each client map inputs into one common space. Every client's prototypes live in that space, so they can be compared and averaged even when the networks behind them differ 2105.00243 app. B.

Variants

Shared per-class prototypes (FedProto)

Each client trains its own model, computes one mean embedding per class it holds, and sends only those vectors to a server. The server averages them per class. Each client then adds a penalty that pulls its embeddings toward the global prototype of their label 2105.00243 eq. 7. Clients may hold different label subsets and different data sizes 2105.00243 §Experiments. Card: Federated Prototype Learning across Heterogeneous Clients.

Reported traffic is 4×10³ parameters per round on MNIST and FEMNIST and 41×10³ on CIFAR10, against 430×10³ and 235×10³ for FedAvg [arxiv:2105.00243 tab. 1]. The exact counting rule behind these figures is not stated 2105.00243 tab. 1.

Heterogeneous client architectures (FedProto-mh)

The same method where clients use different networks. The embedding size must still match across clients 2105.00243 app. B. In the paper's experiments the MNIST and FEMNIST clients differ in the channel count of one convolution layer, and the CIFAR10 clients differ in convolution strides 2105.00243 §Experiments. Accuracy is close to the single-architecture variant on every dataset 2105.00243 tab. 1.

Prototype contrastive loss in federated training (FedProc)

Each client trains on a loss that mixes a prototype contrastive term with cross-entropy. The weight on the contrastive term is α = 1 − t/T in round t of T 2109.12273 eq. 2 2109.12273 alg. 1 l.21. The contrastive term is a softmax over cosine similarities to the class prototypes, with the sample's own class as the positive 2109.12273 eq. 4. The server averages prototypes uniformly over clients and averages models weighted by dataset size 2109.12273 alg. 1 l.9-10. As the algorithm is written, every client therefore runs the same architecture. Author-reported top-1 accuracy with 10 clients on CIFAR-10, CIFAR-100 and Tiny-ImageNet is 70.7%, 74.6% and 35.4%, against 69.1%, 67.5% and 27.5% for MOON 2109.12273 tab. 2. Card: FedProc: Prototypical Contrastive Federated Learning on Non-IID data.

Two points are left open. The paper does not say whether the positives and negatives come from the server average of the previous round or from the client's own class means 2109.12273 §3.3 2109.12273 alg. 1 l.18. Each round also adds a forward pass over the full local dataset to compute prototypes 2109.12273 alg. 1 l.26-30. On CIFAR-10 a round takes 11m57s, against 8m34s for FedAvg 2109.12273 tab. 5.

Nearest class mean with drift compensation (SDC)

In class-incremental learning, the embedding network is trained with a triplet loss and classified by nearest class mean. Prototypes are stored after each task, and no old images are kept 2004.00440 §3.1 2004.00440 §3.2. The embedding moves with each new task, so stored prototypes go stale. SDC estimates how much each old prototype has drifted. It takes a weighted average of how the current task's images moved, with Gaussian weights that fall off with distance to the old prototype. The bandwidth σ is 0.3, or 0.2 on CIFAR100 2004.00440 §4.1 2004.00440 eq. 10 2004.00440 eq. 11 2004.00440 §5. SDC is applied on top of E-FT, E-LwF, E-EWC or E-MAS 2004.00440 §4.2. Card: Semantic Drift Compensation for Class-Incremental Learning.

Author-reported gains on six-task CUB-200-2011, final task: E-FT rises from 37.4 to 49.3 and E-EWC from 54.2 to 56.4 2004.00440 tab. 1. On the eleven-task CIFAR100 and ImageNet-Subset runs, E-LwF+SDC equals E-LwF on CIFAR100 (46.1 each) and is below it on ImageNet-Subset (50.5 against 51.5) 2004.00440 tab. 5.

The official code applies SDC only in the evaluation script, which replays saved per-task models. Training never reads the prototypes test.py train.py. The paper does not say explicitly whether the compensation feeds back into training 2004.00440 §4.2.

Non-learnable cluster prototypes for segmentation (ProtoSeg)

Each class has K prototypes, each the centre of one sub-cluster of that class's training pixels in embedding space. The sub-clusters come from online Sinkhorn clustering, and each prototype is updated by momentum. Prototypes receive no gradient 2203.15102 §3. A pixel is scored against all prototypes by cosine similarity, and the class score is the best score among that class's prototypes 2203.15102 §3 eq. 5–6. The paper's target is the learned class weights or query vectors of softmax and query-based segmentation heads, which it reads as parametric prototypes 2203.15102 §2. K = 10 is the default; K = 20 is best in the ablation, at 46.5 against 46.4 2203.15102 §5.1 2203.15102 tab. 5. Card: Rethinking Semantic Segmentation: A Prototype View.

Author-reported gains on ADE20K validation are +0.6 to +1.2 mIoU over four backbones. The baselines are the authors' own reproductions with the MMSegmentation recipe 2203.15102 tab. 1 2203.15102 §5.1. On ADE20K with 847 classes, K = 10 scores 19.4 mIoU with 27.37 M parameters, against 17.7 mIoU and 33.96 M for the parametric head 2203.15102 tab. 4. In the ablation on ADE20K with MiT-B2 at 40k iterations, cross-entropy alone scores 45.0 and both prototype loss terms together score 46.4 2203.15102 tab. 5.

The released repository has one configuration, HRNet-W48 on Cityscapes. It has no ADE20K or COCO-Stuff prototype setup hrnet.py.

Prototype baseline in zero-shot trigger detection (gen-arg)

This is a different use. The event-extraction paper's zero-shot trigger model is a TapNet-style projection: each token is scored against class vectors built from a few seed keywords, under a linear-chain CRF [arxiv:2104.05919 §2.2 eq. 5–7]. The paper does not present that model as prototype learning. It reports plain prototype networks (Snell et al.) as a separate baseline. On ACE with no event types seen in training, the plain prototype baseline scores 3.19 trigger F1 (TI) and 2.22 (TC), against 55.19 and 52.10 for the TapKey model 2104.05919 tab. 8. With ten frequent types seen, prototype scores and argument heads are stronger but still below the TapKey variant 2104.05919 tab. 8. Card: Document-Level Event Argument Extraction by Conditional Generation.

When to use which

For FedProto, the regularizer weight λ matters. The paper sweeps it on FEMNIST and finds the best value at 1, with accuracy dropping by about 6 points at larger values 2105.00243 fig. 4. The paper's values are MNIST 1, FEMNIST 1 and CIFAR10 0.1 2105.00243 app. A. The λ sweep is one dataset and the paper does not describe how the other λ values were picked.

Implementation sketch

Core loop of FedProto, in our own wording. Each step names its source. The aggregation rule is the one place where the paper's equation and its code differ, so it is marked. The FedProc, SDC and ProtoSeg cards carry their own pseudocode 2109.12273 alg. 1 2004.00440 §4.1 2203.15102 alg. 1.

# Server: keep one global prototype per class, starting empty
# [arxiv:2105.00243 alg. 1] [code: lib/update.py#L162-L163]
global_protos = {}

# Client: one mean embedding per class it holds  [arxiv:2105.00243 eq. 3]
def local_prototypes(embeddings_by_class):
    return {j: mean(e) for j, e in embeddings_by_class.items()}

# Server: combine each class over the clients that hold it
# [arxiv:2105.00243 eq. 6]. The paper weights each client by its share of
# the class; the released code takes a plain mean. See Contradictions.
def aggregate(client_protos):
    return {j: mean(vs) for j, vs in group_by_class(client_protos).items()}

# Client: one local step, cross-entropy plus a pull toward the global
# prototypes  [arxiv:2105.00243 eq. 7]
def local_loss(logits, embeddings, labels, global_protos, lam):
    ce = cross_entropy(logits, labels)
    # a sample whose class has no global prototype adds no pull
    # [code: lib/update.py#L162-L171]
    pull = mse(embeddings, targets_from(labels, global_protos))
    return ce + lam * pull  # lam: 1 (MNIST, FEMNIST), 0.1 (CIFAR10)
                            # [arxiv:2105.00243 app. A]

# Inference: nearest local prototype under L2  [arxiv:2105.00243 eq. 4]
def predict(x, local_protos):
    return argmin(j, l2(embed(x), local_protos[j]))

Contradictions

Within the paper, the FedAvg cell for MNIST with stdev 2 and n=3 is printed as 95.04±6.48 in the main table 2105.00243 tab. 1 and as 91.40±6.48 in the appendix 2105.00243 tab. 2. Neither value is preferred here.

Eq. 6 divides by the number of contributing clients on top of count-based weights 2105.00243 eq. 6, which would shrink global prototypes toward zero. The released code takes an unweighted mean instead utils.py. Which of the two the reported numbers come from is not stated.

Paper and code also differ on the regularizer. The main text uses a distance between prototypes 2105.00243 §Methodology, the appendix uses a squared distance between each sample's embedding and the global prototype 2105.00243 app. B, and the code uses mean squared error over that same pairing update.py. The code's running accuracy bookkeeping may also change the spreads reported in the table update.py 2105.00243 tab. 1. These are not disagreements between papers, and this page does not resolve them.

ProtoSeg's released numbers differ from its paper. The README reports 80.1 mIoU for the prototype model against 79.0 for HRNet, both at 80K iterations and batch size 8 README.md. The paper reports 81.1 against 80.4 on Cityscapes validation, at 160k iterations 2203.15102 tab. 2 2203.15102 §5.1. This page reports the paper's numbers and does not resolve the gap.

Across papers, the numbers are not directly comparable. FedProc takes its baseline rows from other papers and does not mark which were re-run 2109.12273 §4. Its CIFAR10 setting (10 clients, Dirichlet split) also differs from FedProto's (20 clients, n classes per client) 2109.12273 tab. 1 2105.00243 tab. 1. ProtoSeg's segmentation baselines are its own reproductions, and its ADE20K gains are not comparable with FedProto's or SDC's tables.

The word "prototype" is used for different objects. The federated and class-incremental papers use a class mean of embeddings 2105.00243 eq. 3 2004.00440 §3.1. ProtoSeg uses K cluster centres per class, found by clustering and not trained by gradient 2203.15102 §3. The survey's deep prototype networks use learned prototypes that are compared with image patches 2103.11251 §4. The event-extraction paper's baseline is a Snell-style prototype network over class vectors for trigger detection [arxiv:2104.05919 §4.4]. Results for one kind do not transfer to the others, and this page keeps them apart.

Papers

• Federated Prototype Learning across Heterogeneous Clients: per-class mean embeddings shared between federated clients, with a heterogeneous-architecture variant and communication counts per round.
• FedProc: Prototypical Contrastive Federated Learning on Non-IID data: server-averaged class prototypes in a cosine-based contrastive loss for federated training on non-IID image data.
• Semantic Drift Compensation for Class-Incremental Learning: nearest-class-mean classification with stored prototypes shifted by an estimated embedding drift, with no old images kept.
• Rethinking Semantic Segmentation: A Prototype View: non-learnable, clustered prototypes per class replace learned class weights in segmentation heads, with nearest-prototype prediction.
• Document-Level Event Argument Extraction by Conditional Generation: a prototype-network baseline for zero-shot trigger detection, compared with the TapNet-style trigger model the paper uses; the trigger code is not released.
• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges: a survey of interpretable model classes; its case-based section covers deep prototype networks as one open challenge area.