wikillm

Paper · 2019-06-14 · cs.LG, stat.ML

Variational Federated Multi-Task Learning

VIRTUAL trains one personal model per client in federated learning, with a shared server posterior over common weights. Each client refines its own factor against a tempered cavity posterior and returns the change of its server factor; the server multiplies the changes in. Author-reported: best in 8 of 12 Table II rows against FedAvg and FedProx, on MLP, CNN and LSTM tasks.

Federated learningMulti-task learningPersonalized federated learningVariational inferencefederated-learningmulti-taskvariational-inference

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Phones, hospitals or sensors each hold their own data, and their data look different from device to device. A single shared model trained by averaging everyone's updates (the usual federated setup) tends to fit that mix poorly. This paper gives every device its own model, and keeps one shared model on the server that links them.

The shared part is not stored as a single set of numbers. It is stored as a "belief" about each weight: a centre plus a sense of how sure the model is about it. Devices train their own model and send back only how their part of that belief changed. The server multiplies those changes into its belief, so devices that have not been picked in a round are still reflected in it.

The method is built from approximate Bayesian inference. It is meant for ordinary deep networks, not only for the simple convex models that earlier federated multi-task methods needed.

TL;DR

  • VIRTUAL is a federated multi-task method for non-convex networks: a server posterior over shared weights plus a private posterior per client.
  • Clients send the change of their server factor (a ratio of Gaussians), and the server multiplies these changes into its posterior.
  • Reported on six real federated datasets, with MLP, CNN and LSTM models. Author-reported; the paper compares only with FedAvg and FedProx.
  • The official code is a TensorFlow reimplementation with no LICENSE file. Several settings differ from the paper (see Paper vs code).

Problem

Federated learning with one shared model (FedAvg) degrades when client data are strongly non-IID §I. Earlier federated multi-task methods (for example MOCHA) handle per-client models only for convex models §III. The paper asks for per-client models on non-convex networks, with the server seeing only an aggregated update and never an individual client's factor §II-B.

Method

Setup. There are K clients with datasets D_i of N_i labelled examples, and one server. The server holds shared weights θ, and client i holds private weights φ_i. Each client is a discriminative model, with a factorised likelihood over its examples §II-A fig. 1.

Target posterior. The joint posterior over all weights, given all datasets, is proportional to the product of the per-client posteriors divided by the prior raised to the power K − 1 eq. 1.

Approximation. The exact posterior is intractable. The paper uses an expectation-propagation-like scheme: the proxy posterior factorises into one server factor s_i and one client factor c_i per client eq. 2 §II-B. One factor is refined per step, and the refinement minimises a variational free energy [prop. 1] eq. 3.

Free energy. It has three parts eq. 3 app. C:

  • a KL term that pulls the new server factor, multiplied by the cavity, towards the prior tempered by 1/K, multiplied by the same cavity;
  • a KL term that pulls the private client factor towards the client prior, which is the usual Bayes-by-backprop cost;
  • the negative expected log-likelihood of the client's data.

The first KL is the regulariser that keeps the server close to the cavity, so that it learns across clients without forgetting earlier ones §II-B. An optional multiplier β scales both KL terms; it is a tuning knob, not part of the derivation §IV-D.

Parameterisation. For the experiments, every weight has a factorised Gaussian (mean-field) factor for server and client §II-B. The code stores each Gaussian in natural parameters: gamma = mu / sigma² and precision = 1 / sigma². Products of Gaussians become sums of natural parameters, and ratios become differences app. B normal_natural.py.

Round structure. Each round:

  1. a random subset of C clients is chosen;
  2. each chosen client receives the server posterior, computes its cavity, and trains for E epochs on the free energy with SGD and the reparameterisation trick §II-B;
  3. each client sends the ratio of its new and old server factor, which the paper calls Δ_i;
  4. the server multiplies the received ratios into its posterior alg. 1.

Clients that are not chosen keep their private posterior. The server sees only the overall posterior and the aggregated update, never an individual client's factor §II-B.

Damping. Client updates are blended with the previous state using a factor γ in [0, 1], which the paper sets to 1 minus the server learning rate app. D.

Sparse updates. Elements of Δ_i whose signal-to-noise ratio falls below a percentile are set to zero before sending §IV-F.

Pseudocode
# VIRTUAL server round  [alg. 1]
# server_posterior: shared factorised Gaussian over server weights  [§II-B]
# server_factors[i]: client i's current factor on the server  [§II-B]
# client_posteriors[i]: private posterior of client i  [§II-B]

def virtual_round(server_posterior, server_factors, client_posteriors,
                  data, sampled_ids, prior, num_clients, epochs):
    # Each sampled client refines its factors  [alg. 1]
    deltas = []
    for i in sampled_ids:
        old_factor = server_factors[i]
        # Cavity: the server posterior without client i's old factor  [§II-B]
        cavity = server_posterior / old_factor
        # Refine by minimising the free energy, with the prior raised to 1/K
        # [eq. 3] [§II-B]
        server_factors[i], client_posteriors[i] = minimise_free_energy(
            cavity,
            prior ** (1.0 / num_clients),
            client_posteriors[i],
            data[i],
            epochs,
        )
        # The client's update is the ratio of new and old server factor  [§II-B]
        deltas.append(server_factors[i] / old_factor)
    # The server multiplies all received ratios into its posterior  [alg. 1]
    return server_posterior * product(deltas)
Pseudocode
# Local objective for one client  [eq. 3] [§II-B]
# All factors are mean-field Gaussians; optimised by SGD with the
# reparameterisation trick. An optional beta multiplies both KL terms [§IV-D]

def free_energy(cavity, tempered_prior, server_factor, client_posterior,
                client_prior, batch):
    # KL keeping the server factor, inside the cavity, close to the
    # tempered prior inside the cavity  [eq. 3]
    server_kl = kl(server_factor * cavity, tempered_prior * cavity)
    # Bayes-by-backprop KL of the private factor  [eq. 3]
    client_kl = kl(client_posterior, client_prior)
    # Negative expected log-likelihood of the batch  [eq. 3]
    nll = -expected_log_likelihood(batch, server_factor, client_posterior)
    return server_kl + client_kl + nll

Hyperparameters

The paper's own values are listed first; the official code's values follow in the Notes column. Where the paper says "not stated", the code is the only source.

NameValueWhereNotes
Clients per round C10§IV-B app. DCode draws a uniform random subset each round fed_process.py
Local epochs per round E20; one FEMNIST run uses 100§IV-B app. A tab. 4Code grid for FEMNIST: [20, 100] femnist_virtual.json
Batch size B20; Shakespeare 10app. DCode's Shakespeare config grid uses 20 shakespeare_virtual_natural.json
Client optimiserVanilla SGD§IV-B app. DCode uses a custom SGD with per-layer multipliers experiment_utils.py
Client learning rate η_cGrid of 5 log-spaced values; chosen value not statedapp. DCode fixes it: 0.001 (MNIST, VSN, FEMNIST MLP), 0.0001 (FEMNIST CNN), 10.0 (Shakespeare) mnist_virtual.json
Natural-parameter LR multiplierNot statednot statedCode grid: 1e8 to 2e9 (MLP and CNN), 1e9 to 2e10 (Shakespeare). Client multiplier is natural_lr / (η_c · N_i/ΣN) experiment_utils.py
KL multiplier βGrid of 5 log-spaced values; best 1e-5 on FEMNIST§IV-D fig. 2 app. DCode grid: 0, 1e-5, 1e-4, 1e-3, 1e-2 mnist_virtual.json
Server learning rate η_s{0.2, 0.4, 0.6, 0.8, 1.0}app. DSame grid in code mnist_virtual.json
Damping γ1 − η_sapp. DCode default when unset main.py
Prior p(θ)Not statednot statedCode: zero natural mean, precision 1 natural_raparametrization_layer.py
Precision initNot statednot statedCode: untransformed scale ~ N(−4.85, 0.45²) for MLP; sigma = softplus(u), precision = 1 / sigma² tfp_utils.py
Mean initNot statednot statedCode: N(0, 0.5²) for MLP, N(0, 0.65²) for FEMNIST CNN femnist_conv_virtual_natural.json
Rounds TNot statednot statedCode: T = tot_epochs_per_client · K / (C · E); 250 for FEMNIST MLP with 500 epochs main.py
Training budgetNot statednot statedCode: 500 epochs per client (FEMNIST, MNIST), 200 (VSN), 300 (Shakespeare)
FrameworkTensorFlow, TensorFlow Probability§IV-Btfp_utils.py

Architectures §IV-B app. D:

  • MLP: two dense hidden layers of 100 units, ReLU, softmax output. The paper says the Bayesian version uses local reparameterisation; see Paper vs code.
  • FEMNIST CNN: two convolutions (kernel 5, 32 and 64 filters), each followed by max-pooling, then the MLP head.
  • Shakespeare: an 8-dimensional embedding and a two-layer LSTM with 100 units each, with Bayesian LSTMs and Bayesian embeddings; the code uses 256 units (see Paper vs code).

Results

Author-reported. Setup: five federated datasets with 23 to 100 clients tab. 1, C = 10, E = 20, against FedAvg and FedProx. Metrics: max accuracy over rounds, for the server (S) and for the average of all client models (MT) §IV-C §IV-E.

Dataset statistics tab. 1

DatasetKSize (≈)Size/K meanSize/K std
FEMNIST10055k0.5k54
MNIST10060k6000
PMNIST10060k6000
VSN2368k3k559
HAR3015k0.5k56
NLP1001.2m13k11k

Max accuracy at convergence (%) tab. 2 (bold marks the best in each row)

DatasetMetricFedAvgFedProxVirtual
FEMNIST MLPMT94.394.595.7
FEMNIST MLPS90.289.990.9
FEMNIST ConvMT97.397.098.2
FEMNIST ConvS95.595.597.3
MNISTMT96.996.997.4
MNISTS97.697.697.8
PMNISTMT85.985.684.2
PMNISTS48.347.542.5
VSNMT96.296.096.6
VSNS89.689.588.8
HARMT98.999.499.1
HARS94.094.094.3
NLPMT46.146.448.4
NLPS48.148.548.6

Virtual is best in 8 of the 12 rows. The paper reports gains of up to about 2 points (MT) and 1 point (S) on FEMNIST, MNIST and Shakespeare §IV-E. On PMNIST, Virtual is behind FedAvg in both rows §IV-E.

Sparse client updates, FEMNIST MLP tab. 3 (max accuracy, %)

% pruned weightsDelta sizeAccVirtual + FedAvg initVirtual
0%158kMT86.295.6
0%158kS89.490.9
50%79kMT76.394.3
50%79kS81.889.2
75%40kMT78.794.9
75%40kS80.490.8
90%16kMT25.548.5
90%16kS21.020.3

Pruning is applied to the client updates in every round, so the paper calls it harder than pruning a trained network §IV-F. The paper's communication estimate is that 75% sparsity halves communication compared with FedProx, after counting that a Bayesian network stores twice as many parameters §IV-F.

E = 100, FEMNIST tab. 4 app. A (max accuracy, %)

ModelMetricFedAvgFedProxVirtual
MLPMT92.792.893.2
MLPS86.286.687.6
ConvMT97.097.398.2
ConvS95.595.697.3

Effect of β fig. 2 §IV-D: on FEMNIST, β from 1e-6 to 1e-3 does not impair the model. Larger β lets the regulariser dominate. β = 1e-5 gives the best MT loss, and the server loss is unchanged in the adequate range.

Curves fig. 3 §IV-E: the MT metric is smoother than the server metric. It lags early on, because a client is updated only about every K/C rounds, and ends higher than the server metric on most datasets. The exceptions are Shakespeare and the dataset-specific cases the paper discusses §IV-E.

Paper vs code

Source: arXiv v2 LaTeX (updated 2021-02-04). Code: lucori/virtual, the GitHub account of the first author (his personal site lists this paper); commit c3c4abb. The repository has no LICENSE file, so no code is quoted here, only cited.

Search. The paper names no code link. A GitHub search for the method name and title words returned no matching repositories. The first author's GitHub account (lucori) has a repository called virtual, whose README describes this method. The account owner matches the author's personal site, which lists this paper. A commented-out sentence in the arXiv source refers to code in supplementary material; the compiled paper names none.

Discrepancies, each checked by reading the code (none of it was run):

  • Client learning rate. Paper: grid over five values app. D. Code: a single fixed client learning rate per config, and the grid searches a natural-parameter multiplier instead mnist_virtual.json.
  • Client update scaling. Paper: the server multiplies the ratios Δ_i together alg. 1. Code: each sampled client's delta is multiplied by η_s · N_i / ΣN, where ΣN runs over all K clients, not the sampled ones fed_process.py, and the deltas are summed fed_process.py.
  • Damping and server weight both scale the update. The code damps the client's natural parameters and then takes the delta, so the delta is already scaled by γ fed_process.py natural_raparametrization_layer.py. The server then multiplies by η_s · N_i / ΣN. The effective client contribution is therefore proportional to η_s · (1 − η_s) · N_i / ΣN. The paper does not mention the server weight.
  • Prior tempering. Paper: prior raised to 1/K eq. 3. Code: the cavity weight is N_i / ΣN, which equals 1/K only when all clients have the same size virtual_process.py natural_raparametrization_layer.py. MNIST and PMNIST have equal sizes tab. 1; the other datasets do not.
  • KL scaling. Paper: β multiplies both KL terms §IV-D. Code: client KL is divided by N_i, the server KL is not experiment_utils.py. Each client's loss is the batch cross-entropy plus the summed KL terms experiment_utils.py.
  • KL implementation. For two natural-parameter Gaussians, the mean term divides the prior's gamma by the posterior's precision instead of its own. The result is not the standard Gaussian KL unless the two precisions are equal normal_natural.py. This KL is the one registered for the natural Normal type normal_natural.py.
  • Server initialisation. Before training, the code applies the N_i-weighted sum of all K clients' initial parameters to the server fed_process.py. The paper does not describe a server initialisation.
  • Local reparameterisation. Paper: the MLP uses local reparameterisation §IV-B. The MLP and CNN configs use the non-local DenseReparametrizationNaturalShared. Only the Shakespeare output layer is the local variant femnist_virtual.json natural_raparametrization_layer.py.
  • Shakespeare model and batch. Paper: two LSTM layers of 100 units, an 8-d embedding, batch 10, vocabulary 86 §IV-B app. D. Code: two LSTM layers of 256 units, an 8-d embedding over 87 inputs, an 87-way local-reparameterised output, batch 20 shakespeare_virtual_natural.json.
  • Round count. The paper gives no number of rounds. The code derives it from the per-client epoch budget main.py.
  • Reported accuracy. Table II is described as "at convergence" tab. 2. The code records the maximum over rounds of the test accuracy for each metric fed_process.py. The reported numbers are therefore best-round test accuracies.
  • Sparsification. Paper: the SNR of a client's posterior weights is used to rank weights §IV-F fig. 4. Code: the SNR is computed on the update, |Δgamma| / sqrt(|prec|), and the posterior-SNR version is commented out tfp_utils.py. Sparsity is applied only to non-softmax dense layers natural_raparametrization_layer.py. The config grid also includes 95% pruning, which the paper does not report femnist_virtual_natural_sparse_delta.json.
  • Full η_s grid. The grid includes η_s = 1.0. With the code's damping, γ = 0 and every client's delta becomes zero fed_process.py. The paper does not discuss this case.

Pitfalls

  • Natural-parameter scales. The code's learning-rate multipliers for natural parameters are 1e8 to 2e10 mnist_virtual.json. They are not described in the paper and must be reproduced exactly. Reciprocals in the product step are clipped at 1e15 tfp_utils.py.
  • Memory and state. Simulation keeps one posterior per client, so memory grows linearly with K. A real device keeps its own model, and the server's memory does not grow with K §IV intro.
  • Communication. Each update carries a Gaussian parameterisation (two numbers per weight), not a point estimate. The paper estimates that 75% sparsity gives about a 50% communication saving against FedProx, after the factor of two §IV-F.
  • Damping in natural parameters. The appendix writes damping as a power on densities app. D. For exponential-family factors that is a linear blend of natural parameters, which is what the code does natural_raparametrization_layer.py. Implement the blend on gamma and precision, not on means and variances.
  • Test-set selection. Maxima over rounds on the test set are optimistic. The paper does not state the number of seeds or report variance, and it does not mention a held-out split §IV-E.
  • Unstated values. Prior, mean and precision initialisations, the number of rounds and the number of runs are not stated in the paper.
  • Dataset assumptions. The paper says FEMNIST uses 100 random writers and Shakespeare drops roles without a full batch §IV-A. The data code was not checked here.

Lineage

  • Builds on: Bayes by backprop, expectation propagation and the continual-learning Bayesian formulation by Nguyen et al. §II-B §III. Named only; no cards.
  • Compared against: FedAvg and FedProx §IV-E tab. 2. Named only; no cards.
  • Superseded by: not known from content already in this wiki.

Source and license

Adapted from “Variational Federated Multi-Task Learning” by Luca Corinzia, Ami Beuret, Joachim M. Buhmann (arXiv:1906.06268), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/lucori/virtual at commit c3c4abb43ff2 (none (no LICENSE file in repository)). Any quoted code is shown under that license.