--- kind: concept slug: model-merging title: Model merging summary: >- Merge models with one architecture into one, with no joint training run. Variants: matched averaging (FedMA), Fisher-weighted averaging, a closed-form merge of fine-tunes (Custom Diffusion, about 2 s), SLERP/TIES/DARE on LLM weights (BioMistral), or PAINT, which blends a CLIP model with its one-task fine-tune. A shared-subspace projection compresses many fine-tunes instead. tags: [model-fusion, federated-learning, personalization, diffusion, weight-space, fisher-information, llm, robustness] papers: ["2002.06440", "2212.04488", "2402.10373", "2111.09832", "2512.05117", "2208.05592"] related: [federated-learning, diffusion-models, low-rank-adaptation, domain-adaptation, ensembles, transfer-learning, parameter-efficient-fine-tuning, continual-learning, vision-language-pretraining] status: draft updated_at: 2026-10-19 --- ## Overview Model merging builds one model from several models that share an architecture, by combining their parameters directly rather than training on all the data together. Every variant assumes the inputs are the same architecture with the same parameter layout: FedMA's clients all train one architecture [arxiv:2002.06440 §3], and the Custom Diffusion merge combines fine-tunes of one pretrained model [arxiv:2212.04488 §3.2]. Fisher merging also needs the models to start from a shared initialization [arxiv:2111.09832 §2], and PAINT fine-tunes from the zero-shot weights it later blends back with [arxiv:2208.05592 §2]. The six papers here solve different problems with the same idea: - **Federated averaging.** Clients train on private data and send weights. A plain average is wrong when hidden units sit in different orders on different clients, so the server first matches units [arxiv:2002.06440 §2]. - **Weighted averaging.** A plain average gives every parameter the same weight. Fisher merging weights each parameter by how much the model's predictions depend on it [arxiv:2111.09832 §2.2] [arxiv:2111.09832 eq. 4]. - **Concept composition.** Each concept fine-tunes a few weight matrices of a diffusion model. A closed-form solve combines them so that each concept's own words still produce that concept [arxiv:2212.04488 §3.2 eq. 4]. - **Domain LLMs.** A general chat model is continued on medical text, then combined back with its parent model by interpolating or pruning the weight differences. No further training is run [arxiv:2402.10373 §3.3] [arxiv:2402.10373 §5.3]. - **Patching a zero-shot model.** A CLIP encoder is fine-tuned on one task it handles poorly, then blended back with its zero-shot weights. One mixing coefficient sets how much of the new skill is kept [arxiv:2208.05592 §2]. - **Shared subspace.** Across many fine-tunes of one architecture, each layer's weights share a few principal directions. Storing the coordinates in that basis is cheaper than storing the models, and it does not merge them into one model [arxiv:2512.05117 §2]. The BioMistral merges also keep one architecture: BioMistral 7B is merged with its parent Mistral 7B Instruct v0.1 [arxiv:2402.10373 §3.3]. ## Variants ### Matched averaging (FedMA) Units in a layer can be permuted without changing the function, so averaging unaligned weights mixes unrelated units [arxiv:2002.06440 §2] [arxiv:2002.06440 eq. 1]. FedMA matches neurons, convolution channels or LSTM hidden units across clients with a Bayesian nonparametric assignment (BBP-MAP), averages each matched group, and lets an unmatched unit open a new unit in the shared layer [arxiv:2002.06440 §2.1] [arxiv:2002.06440 eq. 3]. Matching runs one layer per pass: clients retrain the layers above a frozen, matched layer, then the next layer is matched [arxiv:2002.06440 §2.3] [arxiv:2002.06440 alg. 1]. Card: [Federated Learning with Matched Averaging](/papers/2002.06440). Author-reported results: VGG-9 on heterogeneous CIFAR-10 with 16 clients reaches 87.53% against 86.29% for FedAvg [arxiv:2002.06440 tab. 1]. An LSTM on Shakespeare with 66 clients reaches 49.07% against 46.63% [arxiv:2002.06440 tab. 2]. The global model grows, to 1.11x the client size for VGG-9 and 1.06x for the LSTM [arxiv:2002.06440 tab. 1] [arxiv:2002.06440 tab. 2]. ### Fisher-weighted averaging Plain averaging gives each parameter the same weight. Fisher merging weights each parameter of each model by that model's diagonal Fisher information, which the paper uses as the posterior precision under a Laplace approximation [arxiv:2111.09832 §2.2] [arxiv:2111.09832 §2.3]. A parameter that the model's predictions depend on keeps its value in the merge, and a parameter they barely depend on gives way to the other models [arxiv:2111.09832 eq. 4]. The Fisher is estimated on training examples, up to 4096 for GLUE [arxiv:2111.09832 §3.3]. Only the shared body is merged; the task heads come from the target model [arxiv:2111.09832 §2.4]. Card: [Merging Models with Fisher-Weighted Averaging](/papers/2111.09832). Author-reported results. For ensembles of five BERT-base checkpoints, Fisher merging beats isotropic merging on RTE, MRPC and SST-2, and is comparable to averaging the five models' predictions at one model's inference cost [arxiv:2111.09832 §3.1] [arxiv:2111.09832 fig. 2]; the numbers appear only in the figure. For WiSE-FT, it gives a better in-distribution and out-of-distribution trade-off, with in-distribution accuracy about 1 point higher at the best average OOD weight [arxiv:2111.09832 §3.2] [arxiv:2111.09832 fig. 3]. For domain adaptation on RoBERTa-base, the gains are small: ChemProt 83.1 against 82.8 for isotropic merging, ACL-ARC 73.2 against 72.5, and SciERC favours isotropic merging, 81.7 against 81.3 [arxiv:2111.09832 tab. 1]. For an RTE target with an MNLI donor, gradient-based sequential training reaches 77.4 ± 1.6 against 73.2 ± 5.1 for Fisher merging [arxiv:2111.09832 §3.3] [arxiv:2111.09832 tab. A1] [arxiv:2111.09832 tab. A3]. The authors estimate the whole merge at about six times fewer FLOPs than fine-tuning, with the Fisher pass as the largest term [arxiv:2111.09832 §3.3]. ### Closed-form merge of concept fine-tunes (Custom Diffusion) Each concept fine-tunes only the key and value projections $W^k$ and $W^v$ of the cross-attention layers, plus one new token embedding [arxiv:2212.04488 §3.1]. To merge $N$ concepts, each layer's $W^k$ and $W^v$ are solved separately. The solve requires the merged matrix to reproduce every concept's target-word outputs while staying close to the pretrained matrix on about 1000 regularization captions [arxiv:2212.04488 §3.2 eq. 4]. The closed form is given in [arxiv:2212.04488 eq. 5]. It takes about 2 s once the single-concept models exist [arxiv:2212.04488 §3.2]. Storage is 75 MB per concept, and a truncated SVD of the update cuts that to about 15 MB with similar metrics at top 60% [arxiv:2212.04488 §4.1] [arxiv:2212.04488 app. C]. Card: [Multi-Concept Customization of Text-to-Image Diffusion](/papers/2212.04488). Joint training of all concepts is the paper's other route: one value $\mathrm{V}^*_i$ per concept, trained together for 500 steps for two concepts [arxiv:2212.04488 §3.2]. On the paper's multi-concept table, joint training scored higher than the merge on most metrics [arxiv:2212.04488 tab. 1]. The table's "Optimization" row is the closed-form merge, per the card's reading. ### Weight interpolation of LLM checkpoints (SLERP, TIES, DARE) BioMistral 7B is continued pre-training of Mistral 7B Instruct v0.1 on PubMed Central text, and the authors merge it back with that parent at 50% weight each [arxiv:2402.10373 §3.2] [arxiv:2402.10373 §5.3]. Three merge methods are compared [arxiv:2402.10373 §3.3]: - **SLERP** interpolates the two parameter sets along a sphere, which the paper motivates as avoiding the information loss of plain averaging. - **TIES** takes a task vector for each model (its difference from a base), keeps the sparse entries, resolves signs by consensus, and adds the result back to the base. - **DARE** is applied on top of TIES: most delta parameters are set to zero at random and the rest rescaled, which the paper reports keeps or improves performance. Author-reported 3-shot English average over ten medical QA tasks: BioMistral 7B 50.3, Mistral 7B Instruct 51.2, SLERP 55.4, DARE 54.6, TIES 51.1 [arxiv:2402.10373 tab. 2]. After supervised fine-tuning, DARE is the best merge at 59.4 against 57.3 for BioMistral 7B [arxiv:2402.10373 tab. 3]. Across seven machine-translated languages, SLERP is 0.1 to 1.5 points above BioMistral 7B on the 10-task average [arxiv:2402.10373 tab. 9]. The paper's "Ensemble" row combines the two models' answer scores at inference and is not a merged checkpoint [arxiv:2402.10373 §5.3]. Card: [BioMistral](/papers/2402.10373). ### Interpolation with a zero-shot model (PAINT) PAINT patches an open-vocabulary CLIP model for one task in two steps. First, the visual encoder is fine-tuned on that task's training data. Second, the zero-shot and fine-tuned weights are blended as (1 - alpha) × zero-shot + alpha × fine-tuned [arxiv:2208.05592 §2]. The head is the frozen output of CLIP's text tower for each class name, so no class-specific parameter is added and new class names still work [arxiv:2208.05592 §3] [arxiv:2208.05592 app. C]. Freezing the head had a negligible effect on MNIST, EuroSAT and SUN397 over three seeds [arxiv:2208.05592 app. C]. Alpha is chosen on held-out validation sets for the supported task and the patch task [arxiv:2208.05592 §2]. Card: [Patching open-vocabulary models by interpolating weights](/papers/2208.05592). Author-reported results. For ViT-L/14 on nine patch tasks where zero-shot CLIP is weak, accuracy rises by 15 to 60 points, and ImageNet accuracy falls by less than one point [arxiv:2208.05592 §1] [arxiv:2208.05592 §4]. Patching on one of two roughly equal, disjoint class sets from a dataset raised accuracy on the other set by 0.8 to 19.4 points across nine datasets [arxiv:2208.05592 §6] [arxiv:2208.05592 tab. 1]. Patching on one dataset raised accuracy on a related dataset in seven of eight pairs, by 1.1 to 19.2 points. The exception was EuroSAT to RESISC45, at -1.3 [arxiv:2208.05592 §6] [arxiv:2208.05592 tab. 2]. For several patch tasks, the paper compares three strategies [arxiv:2208.05592 §5]: - **Joint.** Merge the patch datasets and patch once. For ViT-L/14 this scored 15.8 points above unpatched, 0.5 below ten specialised models [arxiv:2208.05592 §5]. - **Sequential.** Patch each task in turn, starting from the last patched model. For ViT-B/32, interpolation turned a 4.6-point loss into an 11-point gain in combined accuracy, though forgetting still occurs [arxiv:2208.05592 §5]. - **Parallel.** Fine-tune each task from the zero-shot weights, then blend with one shared alpha. The paper reports this as underperforming the other two [arxiv:2208.05592 §5]. ### Shared subspace projection (Universal Subspace) This is not a merge. It stores a set of same-shape fine-tunes as coordinates in a basis shared across them. For each decomposed mode, the models are stacked, centred, and the leading left singular vectors of the unfolding are kept up to a cumulative-variance threshold. Each model is then its coordinates in that basis [arxiv:2512.05117 §2] [arxiv:2512.05117 alg. 1]. The first and last layers are excluded, because their shapes depend on the task [arxiv:2512.05117 app. B.6]. The paper calls the models rebuilt from the basis "Universal" variants [arxiv:2512.05117 §4]. Card: [The Universal Weight Subspace Hypothesis](/papers/2512.05117). Author-reported results. On five held-out ViT models, a 16-dimension subspace reaches 94.1% in-distribution against 94.4% for full training, and 87.8% OOD against 91.3% [arxiv:2512.05117 §4]. Storage falls by about 19x for 500 Mistral LoRAs and by up to 100x for ViTs [arxiv:2512.05117 §4] [arxiv:2512.05117 app. C.2]. Training only the coefficients of a shared basis reaches 90.1% on CIFAR-100 with 10K trainable parameters, against 92.8% for full fine-tuning of 86M parameters [arxiv:2512.05117 §4]. The card does not report a merging result for this method. ## When to use which | Situation | Use | Why | |---|---|---| | Clients keep data private, all run one CNN or LSTM, data is heterogeneous | Matched averaging (FedMA) | Higher accuracy than FedAvg and FedProx in the paper's two settings [arxiv:2002.06440 tab. 1] [arxiv:2002.06440 tab. 2] | | Transformer or other architecture | Not covered | The paper's experiments use VGG-9 and an LSTM [arxiv:2002.06440 §3] | | Limited communication rounds | Matched averaging, with care | One pass needs one round per layer [arxiv:2002.06440 §2.3]. Reported: 11 FedMA passes against 99 FedAvg rounds on VGG-9 [arxiv:2002.06440 §3] | | Global model size is a hard limit | FedAvg or a fixed-size method | FedMA's global model grows, 1.11x for VGG-9 [arxiv:2002.06440 tab. 1] | | Models share an initialization, and a pass over training data for the Fisher is affordable | Fisher-weighted averaging | Beats isotropic averaging in the authors' ensembling and WiSE-FT runs [arxiv:2111.09832 §3.1] [arxiv:2111.09832 §3.2]. Domain-adaptation gains are small [arxiv:2111.09832 tab. 1] | | No Fisher budget, or the paper's plain baseline is enough | Plain averaging (isotropic) | The Fisher pass is the most costly step [arxiv:2111.09832 §2.3]. Isotropic merging is the paper's comparison point [arxiv:2111.09832 §2.1] | | Donor is MNLI for an RTE target, and gradient training is affordable | Gradient training | 77.4 against 73.2 for Fisher merging on RTE [arxiv:2111.09832 tab. A1] [arxiv:2111.09832 tab. A3] | | Donor task is related and no gradient training budget | Fisher merge with the donor | Merging did not hurt in the paper's runs, where gradient intermediate training hurt on some donors [arxiv:2111.09832 fig. 4] | | Concepts were fine-tuned separately and need to be combined in seconds | Closed-form merge | About 2 s per merge, no training [arxiv:2212.04488 §3.2] | | All concepts can be trained together | Joint training | Higher scores on most multi-concept metrics [arxiv:2212.04488 tab. 1] | | Concepts arrive one at a time | Merge, not sequential fine-tuning | Sequential training forgot the first concept [arxiv:2212.04488 §4.2] | | Domain-tuned LLM and its parent share an architecture, no training budget for a merged model | SLERP or DARE | Highest 3-shot English averages after merging (55.4 and 54.6 against 50.3 for BioMistral 7B) [arxiv:2402.10373 tab. 2] | | Fine-tuned LLM will be used after supervised fine-tuning | DARE | Best merge after SFT, 59.4 [arxiv:2402.10373 tab. 3] | | Need a multilingual medical model | SLERP | Above BioMistral 7B in all seven languages, by 0.1 to 1.5 points [arxiv:2402.10373 tab. 9] | | Need one answer distribution from two models at inference | Ensemble, not a merge | The paper's Ensemble row is not a merged checkpoint [arxiv:2402.10373 §5.3] | | Many fine-tunes of one architecture must be stored | Shared subspace projection | About 19x less for 500 LoRAs, up to 100x for ViTs [arxiv:2512.05117 §4]. The authors say the approximation is worse with few models [arxiv:2512.05117 app. B.2] | | A new task must be learned with a few parameters on a shared basis | Coefficient-only training | 90.1% with 10K trainable parameters on CIFAR-100, against 92.8% for full fine-tuning [arxiv:2512.05117 §4] | | Zero-shot open-vocabulary model is weak on one task, and free-text class names must keep working | PAINT | Frozen text-derived head adds no parameters [arxiv:2208.05592 §3]. ViT-L/14 gains 15 to 60 points on nine tasks, ImageNet drops by less than one point [arxiv:2208.05592 §4] | | Patch data for all tasks is available at once | Joint patching | 15.8 points above unpatched, 0.5 below ten specialised models for ViT-L/14 [arxiv:2208.05592 §5] | | Patch tasks arrive one after another | Sequential patching with interpolation | Interpolation turned a 4.6-point loss into an 11-point gain for ViT-B/32. Forgetting still occurs [arxiv:2208.05592 §5] | | Patch tasks are trained on separate machines | Parallel patching with one shared alpha | Can use distributed hardware. The paper's comparison ranks it below joint and sequential [arxiv:2208.05592 §5] | ## Implementation sketch Matched averaging for one layer, in our own words. The paper's BBP-MAP cost terms are not written out; this sketch keeps them as a black box [arxiv:2002.06440 §2.1]. ```pseudocode # Matched averaging for one layer [arxiv:2002.06440 §2.1] [eq. 3] # units[j]: one row per unit of client j's layer # Output: shared layer, perm[j] maps client j's units to its rows def match_layer(units, cost, open_cost): # start from the largest client's units as the global layer global_units = units[largest_client] for j in other_clients: # one assignment per client: match to a global unit, or open # a new one at open_cost [eq. 3] assign = assign_with_opening(cost(units[j], global_units), open_cost) global_units, perm[j] = absorb(global_units, units[j], assign) # several sweeps of remove-and-rematch improve the matching # [arxiv:2002.06440 §2.1] return global_units, perm ``` Fisher-weighted merge of one parameter tensor. The diagonal Fisher of each model is estimated first, on that model's training examples [arxiv:2111.09832 eq. 2]. ```pseudocode # Fisher-weighted merge of one parameter tensor [arxiv:2111.09832 eq. 4] # params[m], fisher[m]: same shape; coeffs[m] >= 0, summing to 1 # Output: merged tensor; the paper's eq. 4 has no normalisation term def fisher_merge(params, fisher, coeffs): # weight of model m at each element: its coefficient times its Fisher weight = [c * f for c, f in zip(coeffs, fisher)] # weighted average, element by element numer = sum(w * p for w, p in zip(weight, params)) # all-near-zero Fisher falls back to the target: see Pitfalls denom = sum(weight) return numer / denom ``` Closed-form merge of one projection (K or V) in one cross-attention layer. Each layer and each projection is solved on its own [arxiv:2212.04488 §3.2]. ```pseudocode # Closed-form merge of one projection [arxiv:2212.04488 §3.2] [eq. 4] [eq. 5] # W0: (o, d) pretrained; W[n]: (o, d) fine-tuned for concept n # C_n: (s_n, d) text features of concept n's target words # R: (s_reg, d) text features of regularization captions def merge_projection(W0, W, C_list, R): # stack every concept's target words, one row per word [eq. 4] C = concat_rows(C_list) # (s, d) # what each fine-tuned model outputs for its own words [eq. 4] V = concat_cols([W[n] @ C_list[n].T for n in range(N)]) # (o, s) # D = C (R^T R)^-1 weights the regularization directions [eq. 5] D = C @ inverse(R.T @ R) # (s, d) # correction that enforces W C^T = V [eq. 5] v = (V - W0 @ C.T) @ inverse(D @ C.T) # (o, s) return W0 + v @ D # (o, d) ``` ## Contradictions No disagreement between the papers, which address different problems. Within the Fisher paper, merging did not hurt its intermediate-task runs, but gradient training won for the MNLI donor on RTE [arxiv:2111.09832 fig. 4] [arxiv:2111.09832 tab. A1]. The authors' claim of a cheaper alternative holds for some donors, not all [arxiv:2111.09832 §3.3]. Sequential adaptation looks different in two papers: Custom Diffusion's sequential fine-tuning forgot the first concept [arxiv:2212.04488 §4.2], while PAINT's sequential patching with interpolation gained accuracy for ViT-B/32 [arxiv:2208.05592 §5]. These differ in method, model and task, so they do not test the same claim. Gaps between a paper and its released code are listed under Pitfalls. ## Pitfalls - **FedMA averaging rule.** The paper averages matched units uniformly [arxiv:2002.06440 alg. 1]. The card's paper-vs-code check of the released code (commit 4b586a5) found a Bayesian posterior mean instead, which pulls a unit shared by $m$ clients toward the prior by a factor of $m/(m+1)$ when both variances are 1 ([FedMA card](/papers/2002.06440)). Decide which one to reproduce. - **FedMA rounds.** "Rounds" in the paper's comparisons appear to count passes, not single communication rounds. This is the card's inference from the numbers, not something the paper states [arxiv:2002.06440 §3]. - **FedMA and batch norm.** Batch-norm layers were removed from VGG-9. The paper does not say how running statistics would be matched [arxiv:2002.06440 §3]. - **FedMA and LSTM recurrence.** The LSTM matching cost ignores hidden-to-hidden weights, which are only permuted and averaged afterwards [arxiv:2002.06440 §2.2]. - **Merge conditioning.** $R^\top R$ is $d\times d$, so it needs at least $d$ independent regularization token features. Duplicate target rows make $DC^\top$ singular. The paper only states the non-degeneracy assumption [arxiv:2212.04488 §3.2]. - **Equation 4 shape.** With $W\in\mathbb{R}^{o\times d}$, the constraint needs $V$ as $o\times s$. The printed form suggests $s\times o$ [arxiv:2212.04488 eq. 4]. - **Batch size and learning rate.** Batch 8 is 2 GPUs times 4, and the learning rate $8\times10^{-5}$ comes from scaling the base rate by the total batch. Training on one GPU with the same settings halves both [arxiv:2212.04488 app. E]. - **Real regularization images matter.** With generated instead of retrieved images, KID rises from 20.96 to 34.70 [arxiv:2212.04488 tab. 3]. - **Merged models need the token embeddings.** Each concept's modifier embedding must be loaded too, and target features must be computed with them in place [arxiv:2212.04488 §3.2]. - **Fisher normalisation is a code default, not in the paper.** The released code divides each model's coefficient by the L2 norm of its Fisher. The paper's eq. 4 has no such term, and whether its results used it is not stated ([Fisher card](/papers/2111.09832)). Say which variant you implement. - **Fisher floor differs from the paper.** The paper uses the target value when the Fisher is near zero for all models [arxiv:2111.09832 §2.4]. The code floors only the target's Fisher at 1e-6, so a near-zero target Fisher still gets a small weight, and donor parameters with near-zero Fisher get almost none ([Fisher card](/papers/2111.09832)). - **Fisher merge selects on the reported split.** The coefficient search picks the best merge on data whose scores it reports, which inflates those scores. Reproduce with a held-out split ([Fisher card](/papers/2111.09832)). - **Fisher gains can be within noise.** On ChemProt, Fisher merging is 83.1 ± 0.4 against 82.8 ± 0.4 for isotropic merging [arxiv:2111.09832 tab. 1]. - **Fisher cost and FLOPs.** The diagonal needs roughly the compute of training on N examples, since it needs a per-example gradient for each class [arxiv:2111.09832 §2.3]. The FLOP figures are heuristic estimates, not measured times [arxiv:2111.09832 §3.3]. - **Laplace assumptions.** The diagonal ignores correlations between parameters, and assumes the trained parameters are a local maximum of the posterior [arxiv:2111.09832 §2.3]. The Fisher is a local property, so the method may work worse for parameters that are far apart [arxiv:2111.09832 §2.4]. - **Averages hide task swings.** Per task, merged models move PubMedQA toward the Instruct model's score (SLERP 68.1, DARE 70.0, against 37.6 for BioMistral 7B and 72.2 for Mistral 7B Instruct). TIES reaches 44.3 on it [arxiv:2402.10373 tab. 2]. Check per task, not only on the average. - **Merge weights are a fixed 50/50 in the paper.** The paper states equal weights [arxiv:2402.10373 §5.3]. The SLERP interpolation factor is not stated there; the card's reading of the released configs is a per-layer schedule with a default of 0.5 ([BioMistral card](/papers/2402.10373)). - **DARE definition.** The paper describes DARE as random pruning plus rescaling [arxiv:2402.10373 §3.3]. The released config uses a `dare_ties` method with TIES-style sign handling. Decide which one to reproduce. - **Ensemble is not a merge, and its aggregation differs.** The paper says the ensemble aggregates log probabilities [arxiv:2402.10373 §5.3]; the released code adds per-letter probabilities. Do not carry ensemble results over to a merged model, or from one aggregation to the other. - **Shared subspace is an empirical claim.** The theory is conditional, and it does not prove that the subspace exists [arxiv:2512.05117 §3]. Treat "universal" as "low-rank in these experiments". - **Shared subspace needs many models.** The authors say the approximation is worse with few models [arxiv:2512.05117 app. B.2] [arxiv:2512.05117 app. B.3]. The OOD gap of the Universal ViT, 87.8% against 91.3%, is not analysed [arxiv:2512.05117 §4]. - **Subspace centring and variance threshold.** Algorithm 1 centres the models, but the released HOSVD path does not. The cumulative explained-variance threshold is not normalised in the paper ([Universal card](/papers/2512.05117)). - **LoRA factors are not unique.** The subspace describes the stored A and B factors, not the update BA [arxiv:2512.05117 app. B.5]. - **Subspace storage excludes layers.** The reported savings leave out the first and last layers [arxiv:2512.05117 app. C.2]. Full-model storage needs them added back. - **No license on the Fisher and Universal code.** Neither repository has a LICENSE file, so do not copy code from them ([Fisher card](/papers/2111.09832), [Universal card](/papers/2512.05117)). - **No license on the LLM merge code.** The BioMistral repository has no license file, so do not copy its merge configs or scripts; the card does not quote them ([BioMistral card](/papers/2402.10373)). - **PAINT alpha is per model.** The paper picks alpha on validation data. In the Avg column of its single-task tables, ViT-B/32 peaks near alpha 0.5 to 0.6 and ViT-L/14 near 0.65 to 0.70. Treat these as examples and choose alpha on your own validation data [arxiv:2208.05592 app. O]. - **PAINT code does not select alpha.** The patch script saves a model for every alpha in its list, and the choice is left to you [code: src/patch.py#L42-L60]. - **PAINT code patches one dataset per run.** No joint, sequential or parallel multi-task code was found in its `src/` directory [code: src/patch.py#L25-L60]. - **PAINT paper and code defaults differ.** The paper uses learning rate 1e-5, 200 warm-up steps and 2000 iterations. The code defaults are 1e-3, 500 warm-up steps and 10 epochs, and it clips the gradient norm at 1.0, which the paper does not mention [arxiv:2208.05592 §3] [code: src/args.py#L49-L75] [code: src/finetune.py#L95]. - **PAINT head prompts.** The code averages several prompt templates per dataset for the head. The paper does not say which prompts it used [code: src/heads.py#L13-L49]. - **Sequential PAINT still forgets.** Interpolation reduces forgetting but does not remove it, and forgetting can compound across steps [arxiv:2208.05592 §5] [arxiv:2208.05592 §9]. - **Licensing.** The released Custom Diffusion code is under a non-commercial license. The sketch above is written from the paper's equations; do not replace it with code copied from that repository ([Multi-Concept card](/papers/2212.04488)). ## Papers - [Federated Learning with Matched Averaging](/papers/2002.06440): matches neurons, channels and LSTM hidden units across clients layer by layer, then averages the matched units. - [Merging Models with Fisher-Weighted Averaging](/papers/2111.09832): weights each parameter by its diagonal Fisher information when averaging models from one initialization, for ensembles, WiSE-FT and donor-task merges. - [Multi-Concept Customization of Text-to-Image Diffusion](/papers/2212.04488): fine-tunes only cross-attention K/V per concept, then merges concepts by a closed-form constrained least-squares solve. - [BioMistral: Open-Source Pretrained Large Language Models for Medical Domains](/papers/2402.10373): continued pre-training of Mistral 7B Instruct on PubMed Central, then SLERP, TIES and DARE merges with the parent model, evaluated on medical QA. - [Patching open-vocabulary models by interpolating weights](/papers/2208.05592): fine-tunes a CLIP visual encoder on one task, then interpolates with the zero-shot weights; also tests joint, sequential and parallel multi-task patching. - [The Universal Weight Subspace Hypothesis](/papers/2512.05117): stores many same-architecture fine-tunes as coordinates in a shared per-layer basis, with storage and coefficient-only adaptation results.