---
kind: concept
slug: hypernetwork
title: Hypernetwork
summary: >-
  A network whose output is the weights of another network. Two variants here: a server-side hypernetwork maps a learned client embedding to that client's weights, and only weight changes travel back (federated); a document hypernetwork writes a rank-8 LoRA adapter from a document's activations in one forward pass. Both trade up-front training for fast, per-input weights.
tags: [hypernetwork, federated-learning, personalization, long-context, fine-tuning]
papers: ["2103.04628", "2602.15902"]
related: [federated-learning, personalized-federated-learning, context-distillation, low-rank-adaptation]
status: draft
updated_at: 2026-10-19
---

## Overview

A hypernetwork is a network whose output is the weights of another network,
called the target [arxiv:2103.04628 §2.3]. Two design choices define a variant.
The first is the input: a learned embedding per client in the federated card
[arxiv:2103.04628 §3.2], or the activations a frozen LLM computes over a
document in the document card [arxiv:2602.15902 §3.2]. The second is the
output: the full target weights in the first case, or a low-rank adapter
(LoRA) for the frozen model in the second [arxiv:2602.15902 §3.2].

How the training signal reaches the hypernetwork also differs. The federated
variant sends the target's weight change back and maps it into the
hypernetwork with the chain rule [arxiv:2103.04628 §3.2]. The document variant
is trained so the adapted model matches a teacher that reads the document in
its prompt [arxiv:2602.15902 §3.1]. In both, the hypernetwork is trained
ahead of time, and new weights are then produced by a forward pass rather than
by training the target again.

## Variants

### Personalized federated hypernetwork (pFedHN)

A server-side hypernetwork produces the full personal network of each client
from its embedding. All clients share one set of hypernetwork parameters φ, so
one set of parameters holds a family of personal models [arxiv:2103.04628 §3.2].
Each client's embedding is trained with the rest of the hypernetwork. The
hypernetwork is never sent to clients; only target-sized vectors move
[arxiv:2103.04628 §3.2].

The paper's hypernetwork is an MLP with three hidden layers of 100 units and
one output head per target tensor [arxiv:2103.04628 app. B] [arxiv:2103.04628 §5].
Each round, one client's weights are generated, the client runs K local steps
(the paper uses K = 50) and returns the change [arxiv:2103.04628 §5]
[arxiv:2103.04628 alg. 1]. Card:
[Personalized Federated Learning using Hypernetworks](/papers/2103.04628).
Author-reported, on CIFAR10 with 10 clients, it reaches 90.83 ± 1.56 against
89.11 ± 2.66 for the best baseline in that column, LG-FedAvg
[arxiv:2103.04628 tab. 1].

### Personal classifier variant (pFedHN-PC)

The hypernetwork produces only the feature extractor. Each client keeps its own
output layer, which is trained locally and never sent [arxiv:2103.04628 §3.3]
[arxiv:2103.04628 eq. 4]. The paper uses this when clients solve different
tasks, for example with different numbers of classes [arxiv:2103.04628 §3.3].
Author-reported, on Omniglot with 50 clients, it reaches 81.89 ± 0.15 against
76.46 ± 0.17 for Per-FedAvg, the best baseline there [arxiv:2103.04628 tab. 1].

### Document-to-LoRA hypernetwork (D2L)

A frozen LLM reads a document once, and the hypernetwork writes a rank-8 LoRA on
the down-projection of each MLP block [arxiv:2602.15902 §5]. Its input is the
per-layer activations of the frozen model, cross-attended by learned latent
queries [arxiv:2602.15902 §3.2]. Documents longer than 8192 tokens are split
into chunks, each chunk yields its own adapter, and the adapters are
concatenated along the rank dimension, so the output shape does not depend on
the length [arxiv:2602.15902 §3.2] [arxiv:2602.15902 §5].

Training is a teacher-student objective. The student is the base model with the
generated adapter and no context. The teacher is the same base model with the
context in its prompt. Queries and answers are generated by the model itself
over a corpus of about 3.2 million contexts, and the teacher's top-16 token distributions
are stored as targets [arxiv:2602.15902 §3.1] [arxiv:2602.15902 app. B.1]. The
paper also gives a two-stage schedule, with random chunking in the second stage
[arxiv:2602.15902 app. B.2]. Card:
[Doc-to-LoRA: Learning to Instantly Internalize Contexts](/papers/2602.15902).

Author-reported, on 2WikiMultihopQA, batched D2L reaches normalized performance
0.857 (ROUGE-L F1 relative to the base model with the full context) at 0.209 s
per update (± 0.123). Context distillation with the oracle query reaches 0.901
at 40.171 s [arxiv:2602.15902 tab. 1]. Iterative mode trades latency for update
memory: 0.844 at 0.551 s, with 3.791 GB peak against 7.820 GB for the oracle
[arxiv:2602.15902 tab. 1].

## When to use which

These rows follow what the papers' setups and results support. Numbers are the
authors' and are not independently checked.

| Situation | Use | Why |
|---|---|---|
| Clients share one output space and each needs a full personal network | pFedHN | The hypernetwork generates every target tensor [arxiv:2103.04628 §3.2] |
| Clients have different output heads or class counts | pFedHN-PC | Output layer stays local and is never generated [arxiv:2103.04628 §3.3] |
| Link bandwidth is the constraint | Either | Each round sends a target-sized vector, not the hypernetwork, so the cost depends on the target size [arxiv:2103.04628 §3.2] |
| Several target sizes must be served | pFedHN | One hypernetwork produced small, medium and large targets, and the authors report it beat per-size baselines [arxiv:2103.04628 §5.2] [arxiv:2103.04628 tab. 2] |
| A new client joins after training | pFedHN, with a frozen hypernetwork | Only a new embedding is optimized; the paper reports better generalization than its baselines, with step count and initialization not stated [arxiv:2103.04628 §5.3] [arxiv:2103.04628 fig. 2] |
| Limited local compute per round | Plan for K local steps | The authors report K = 50 beat a single step, but each round still runs K steps on the client [arxiv:2103.04628 app. C] |
| Documents or facts change often and per-document fine-tuning is too slow | D2L, batched | One forward pass per document; the oracle takes about 40 s per update [arxiv:2602.15902 §1] [arxiv:2602.15902 tab. 1] |
| Long contexts where prompt length and memory are the cost, and some quality loss is acceptable | D2L, iterative | Lowest update memory of the D2L modes, 3.791 GB [arxiv:2602.15902 tab. 1] |
| Tight per-document budget with few generated queries | D2L over context distillation | SQuAD: 0.866 at 0.086 s against 0.650 at 631 s for CD with 100 generated queries [arxiv:2602.15902 tab. 4]. On MultiFieldQA, CD with 25 queries scores higher than D2L batched (0.528 against 0.481) [arxiv:2602.15902 tab. 9] |
| Best quality and per-query training is acceptable | Context distillation with the oracle query | Highest reported quality, but it trains on the target query, so it is a reference rather than a deployable option [arxiv:2602.15902 §5] [arxiv:2602.15902 tab. 1] |

Limits of the document variant. With an unrelated context, D2L answers worse
than the base model: on SQuAD with a system prompt as the context, it scores
0.096 against 0.201 for the base model [arxiv:2602.15902 tab. 8]. The paper
suggests adding irrelevant queries as a remedy and does not test it
[arxiv:2602.15902 app. C.4]. The paper also reports weaker results on
Mistral-7B-Instruct-v0.2 than on the main model, without an explanation
[arxiv:2602.15902 app. E].

## Implementation sketch

Language-agnostic core of one federated server round, following the paper's
loop [arxiv:2103.04628 alg. 1]. The update sign is the part to check; see
Contradictions.

```pseudocode
# One round: the server picks a client, the client trains, the server updates  [alg. 1]
def hypernet_round(phi, emb, client, eta, K, alpha):
    # Server generates the personal weights from the client embedding  [§3.2]
    theta = h(emb[client], phi)
    # Client runs K local mini-batch steps from theta  [alg. 1] [app. C]
    theta_k = theta
    for _ in range(K):
        theta_k = theta_k - eta * grad_loss(theta_k, client.minibatch())
    # Server takes the chain-rule step for phi and the client embedding  [§3.2]
    # direction: descent on the surrogate loss; sign see Contradictions
    delta = theta - theta_k
    g_phi, g_emb = vjp(h, (emb[client], phi), delta)  # Jacobian of theta wrt inputs
    phi = phi - alpha * g_phi                         # outer lr: grid in [app. B]
    emb[client] = emb[client] - alpha * g_emb
    return phi, emb
```

Variant notes: for pFedHN-PC, `h` returns only the feature-extractor weights,
and the client's output layer stays out of the update [arxiv:2103.04628 §3.3].
The paper does not state the batch size, momentum or clipping, so a rebuild
must choose them itself [arxiv:2103.04628 app. B].

For D2L, the input to the hypernetwork is the frozen model's activations over
the document, and the output is LoRA factors A and B for each layer, produced
in one pass and split into chunks [arxiv:2602.15902 §3.2]. The update is the
student-teacher divergence over the stored teacher targets, not the
weight-change rule above [arxiv:2602.15902 §3.1]. The main-run learning rates
and optimizer are not stated in the paper [arxiv:2602.15902 app. B.2].

## Contradictions

- **Update sign.** The printed Algorithm 1 and the paper's surrogate argument
  in §3.2 give opposite signs for the weight change. The card's Paper vs code
  section describes the official code as following §3.2, and the pseudocode
  above follows that reading [arxiv:2103.04628 alg. 1] [arxiv:2103.04628 §3.2].
- **Claimed margins.** The text says the method gains 2% to 10% over every
  competitor on the CIFAR experiments [arxiv:2103.04628 §5.1]. Computed from
  Table 1 against the best baseline per column, the gap is +1.7 to +2.7 on
  CIFAR10 and +3.3 to +10.0 on CIFAR100, and −4.4 on Omniglot
  [arxiv:2103.04628 tab. 1]. Table 2 gaps are +2.8 to +6.6 against the text's
  4% to 8% [arxiv:2103.04628 §5.2] [arxiv:2103.04628 tab. 2]. The text and the
  table do not agree; the table numbers are our arithmetic.
- **Pair-wise comparison.** On Table 1, pFedHN-PC is higher than pFedHN in
  every column except CIFAR100 with 100 clients (52.4 against 53.24)
  [arxiv:2103.04628 tab. 1]. The paper gives no single rule for when the
  extra personal head is worth its local cost.
- **SQuAD relative performance.** The text puts D2L at 82.5% of the in-context
  upper bound on SQuAD [arxiv:2602.15902 §5.1]. The tables list 0.814 for that
  configuration [arxiv:2602.15902 tab. 2], which is about 94% of the 0.8692
  reference with truncated context [arxiv:2602.15902 tab. 11]. The gap is not
  resolved.
- **Not comparable across variants.** The pFedHN numbers are classification
  accuracy on CIFAR and Omniglot [arxiv:2103.04628 tab. 1]; the D2L numbers are
  normalized ROUGE-L on document QA [arxiv:2602.15902 tab. 1]. This page makes
  no ranking claim between the two.

## Papers

- [Personalized Federated Learning using Hypernetworks](/papers/2103.04628):
  introduces the server-side hypernetwork with client embeddings for federated
  personalization, and the personal classifier variant.
- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](/papers/2602.15902):
  a hypernetwork that writes a rank-8 LoRA for a frozen LLM from a document's
  activations, trained to match context distillation.