Hypernetwork

Overview

A hypernetwork is a network whose output is the weights of another network, called the target 2103.04628 §2.3. Two design choices define a variant. The first is the input: a learned embedding per client in the federated card 2103.04628 §3.2, or the activations a frozen LLM computes over a document in the document card 2602.15902 §3.2. The second is the output: the full target weights in the first case, or a low-rank adapter (LoRA) for the frozen model in the second 2602.15902 §3.2.

How the training signal reaches the hypernetwork also differs. The federated variant sends the target's weight change back and maps it into the hypernetwork with the chain rule 2103.04628 §3.2. The document variant is trained so the adapted model matches a teacher that reads the document in its prompt 2602.15902 §3.1. In both, the hypernetwork is trained ahead of time, and new weights are then produced by a forward pass rather than by training the target again.

Variants

Personalized federated hypernetwork (pFedHN)

A server-side hypernetwork produces the full personal network of each client from its embedding. All clients share one set of hypernetwork parameters φ, so one set of parameters holds a family of personal models 2103.04628 §3.2. Each client's embedding is trained with the rest of the hypernetwork. The hypernetwork is never sent to clients; only target-sized vectors move 2103.04628 §3.2.

The paper's hypernetwork is an MLP with three hidden layers of 100 units and one output head per target tensor 2103.04628 app. B 2103.04628 §5. Each round, one client's weights are generated, the client runs K local steps (the paper uses K = 50) and returns the change 2103.04628 §5 2103.04628 alg. 1. Card: Personalized Federated Learning using Hypernetworks. Author-reported, on CIFAR10 with 10 clients, it reaches 90.83 ± 1.56 against 89.11 ± 2.66 for the best baseline in that column, LG-FedAvg 2103.04628 tab. 1.

Personal classifier variant (pFedHN-PC)

The hypernetwork produces only the feature extractor. Each client keeps its own output layer, which is trained locally and never sent 2103.04628 §3.3 2103.04628 eq. 4. The paper uses this when clients solve different tasks, for example with different numbers of classes 2103.04628 §3.3. Author-reported, on Omniglot with 50 clients, it reaches 81.89 ± 0.15 against 76.46 ± 0.17 for Per-FedAvg, the best baseline there 2103.04628 tab. 1.

Document-to-LoRA hypernetwork (D2L)

A frozen LLM reads a document once, and the hypernetwork writes a rank-8 LoRA on the down-projection of each MLP block 2602.15902 §5. Its input is the per-layer activations of the frozen model, cross-attended by learned latent queries 2602.15902 §3.2. Documents longer than 8192 tokens are split into chunks, each chunk yields its own adapter, and the adapters are concatenated along the rank dimension, so the output shape does not depend on the length 2602.15902 §3.2 2602.15902 §5.

Training is a teacher-student objective. The student is the base model with the generated adapter and no context. The teacher is the same base model with the context in its prompt. Queries and answers are generated by the model itself over a corpus of about 3.2 million contexts, and the teacher's top-16 token distributions are stored as targets 2602.15902 §3.1 2602.15902 app. B.1. The paper also gives a two-stage schedule, with random chunking in the second stage 2602.15902 app. B.2. Card: Doc-to-LoRA: Learning to Instantly Internalize Contexts.

Author-reported, on 2WikiMultihopQA, batched D2L reaches normalized performance 0.857 (ROUGE-L F1 relative to the base model with the full context) at 0.209 s per update (± 0.123). Context distillation with the oracle query reaches 0.901 at 40.171 s 2602.15902 tab. 1. Iterative mode trades latency for update memory: 0.844 at 0.551 s, with 3.791 GB peak against 7.820 GB for the oracle 2602.15902 tab. 1.

When to use which

These rows follow what the papers' setups and results support. Numbers are the authors' and are not independently checked.

Limits of the document variant. With an unrelated context, D2L answers worse than the base model: on SQuAD with a system prompt as the context, it scores 0.096 against 0.201 for the base model 2602.15902 tab. 8. The paper suggests adding irrelevant queries as a remedy and does not test it 2602.15902 app. C.4. The paper also reports weaker results on Mistral-7B-Instruct-v0.2 than on the main model, without an explanation 2602.15902 app. E.

Implementation sketch

Language-agnostic core of one federated server round, following the paper's loop 2103.04628 alg. 1. The update sign is the part to check; see Contradictions.

# One round: the server picks a client, the client trains, the server updates  [alg. 1]
def hypernet_round(phi, emb, client, eta, K, alpha):
    # Server generates the personal weights from the client embedding  [§3.2]
    theta = h(emb[client], phi)
    # Client runs K local mini-batch steps from theta  [alg. 1] [app. C]
    theta_k = theta
    for _ in range(K):
        theta_k = theta_k - eta * grad_loss(theta_k, client.minibatch())
    # Server takes the chain-rule step for phi and the client embedding  [§3.2]
    # direction: descent on the surrogate loss; sign see Contradictions
    delta = theta - theta_k
    g_phi, g_emb = vjp(h, (emb[client], phi), delta)  # Jacobian of theta wrt inputs
    phi = phi - alpha * g_phi                         # outer lr: grid in [app. B]
    emb[client] = emb[client] - alpha * g_emb
    return phi, emb

Variant notes: for pFedHN-PC, h returns only the feature-extractor weights, and the client's output layer stays out of the update 2103.04628 §3.3. The paper does not state the batch size, momentum or clipping, so a rebuild must choose them itself 2103.04628 app. B.

For D2L, the input to the hypernetwork is the frozen model's activations over the document, and the output is LoRA factors A and B for each layer, produced in one pass and split into chunks 2602.15902 §3.2. The update is the student-teacher divergence over the stored teacher targets, not the weight-change rule above 2602.15902 §3.1. The main-run learning rates and optimizer are not stated in the paper 2602.15902 app. B.2.

Contradictions

• Update sign. The printed Algorithm 1 and the paper's surrogate argument in §3.2 give opposite signs for the weight change. The card's Paper vs code section describes the official code as following §3.2, and the pseudocode above follows that reading 2103.04628 alg. 1 2103.04628 §3.2.
• Claimed margins. The text says the method gains 2% to 10% over every competitor on the CIFAR experiments 2103.04628 §5.1. Computed from Table 1 against the best baseline per column, the gap is +1.7 to +2.7 on CIFAR10 and +3.3 to +10.0 on CIFAR100, and −4.4 on Omniglot 2103.04628 tab. 1. Table 2 gaps are +2.8 to +6.6 against the text's 4% to 8% 2103.04628 §5.2 2103.04628 tab. 2. The text and the table do not agree; the table numbers are our arithmetic.
• Pair-wise comparison. On Table 1, pFedHN-PC is higher than pFedHN in every column except CIFAR100 with 100 clients (52.4 against 53.24) 2103.04628 tab. 1. The paper gives no single rule for when the extra personal head is worth its local cost.
• SQuAD relative performance. The text puts D2L at 82.5% of the in-context upper bound on SQuAD 2602.15902 §5.1. The tables list 0.814 for that configuration 2602.15902 tab. 2, which is about 94% of the 0.8692 reference with truncated context 2602.15902 tab. 11. The gap is not resolved.
• Not comparable across variants. The pFedHN numbers are classification accuracy on CIFAR and Omniglot 2103.04628 tab. 1; the D2L numbers are normalized ROUGE-L on document QA 2602.15902 tab. 1. This page makes no ranking claim between the two.

Papers

• Personalized Federated Learning using Hypernetworks: introduces the server-side hypernetwork with client embeddings for federated personalization, and the personal classifier variant.
• Doc-to-LoRA: Learning to Instantly Internalize Contexts: a hypernetwork that writes a rank-8 LoRA for a frozen LLM from a document's activations, trained to match context distillation.