Paper · 2026-10-19 · cs.LG, cs.CL
An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs
Controlled PEFT study on three pretrained MoE LLMs (Qwen3-30B-A3B, Gemma-4-26B-A4B, GPT-OSS-20B) on math and intent tasks, with MMLU-Pro and IFEval retention. Adapting expert weights beats attention or bypass adapters; LoRA and DoRA lead; no nonzero balancing coefficient beats zero. A MoE2-LoRA redesign gains 2.4-17.9 upstream points with 17% fewer parameters (author-reported).
Parameter-efficient fine-tuningLow-rank adaptation (LoRA and variants)Mixture of experts (MoE)Expert load balancingfine-tuningmixture-of-expertsloradorapeftload-balancingqwen3
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A mixture-of-experts (MoE) language model is built from many small expert networks, and a router sends each token to only a few of them. Fine-tuning such a model cheaply usually means freezing most of it and training a small add-on, but the usual add-on recipes were designed for ordinary dense models. This paper asks which parts of an MoE model to add to, and how, if you want the model to learn a new task without losing what it already knows.
The authors run a large set of controlled experiments on three MoE models and two new tasks, and they measure both the new-task score and a score for general knowledge and instruction following. Their main finding is that the best place to add trainable parts is inside the experts themselves, with the weights of the experts updated directly. Standard low-rank adapters (LoRA) and their variant DoRA work best among the methods they tried. A load-balancing penalty, which is often added to keep the router spreading tokens evenly, tends to hurt rather than help. Changing one of the authors' earlier methods to follow these rules keeps more general ability while using fewer trainable parameters.
TL;DR
- Controlled study of PEFT on three pretrained MoE LLMs: Qwen3-30B-A3B-Base, Gemma-4-26B-A4B and GPT-OSS-20B, on math reasoning and intent classification §3.2.
- Adapting routed expert weights alone gives the highest downstream score, with competitive general-capability retention; LoRA and DoRA lead among PEFT methods §4.1 §4.3.
- No nonzero auxiliary load-balancing coefficient beats the zero baseline on the Pareto view; the authors recommend zero or about 1e-4 §4.4 app. D.2.
- On Qwen3, a weight-level redesign of MoE2-LoRA raises each of its four upstream scores by 2.4 to 17.9 points while using 17% fewer trainable parameters §5 tab. 1.
- All numbers are author-reported. The code is Apache-2.0 and matches most, but not all, of the paper's protocol (see Paper vs code).
Problem
Parameter-efficient fine-tuning (PEFT) methods were mostly validated on dense models. In an MoE model, the router decides which experts see each token, so a small set of trainable parameters receives uneven, task-dependent gradient signal. The paper states that the community lacks a systematic recipe for applying PEFT to MoE models §1 §2. It asks four design questions: where the adapters go, what granularity suits the MoE modules, which PEFT method to choose, and how much weight the auxiliary balancing penalty should get §1.
The study holds the trainable parameter budget roughly comparable across arms and keeps general-capability retention in view. It also keeps the backbone frozen apart from the chosen adapters §3.2 app. A.
Method
The paper is an empirical study, not a new training method. Its setup is:
- Treat fine-tuning as an additive update to pretrained weights, W' = W + ΔW eq. 2. For LoRA, ΔW = (α/r) B A, with A of shape r × d_in and B of shape d_out × r §3.1.
- Write the MoE layer as a router g followed by a top-k selection, a renormalised weighted sum of expert outputs, and an optional always-on shared expert eq. 1 app. A.
- Evaluate on two downstream tasks: MATH-500 after training on MetaMathQA-R1, and exact-match intent accuracy after training on ESFT_Intent §3.2 app. B.
- Measure general capability with MMLU-Pro and IFEval, using the same checkpoints §3.2.
- Select one learning rate per configuration by mean downstream score over three seeds. General scores never enter the selection §3.2 app. B.2.
The four design axes are:
- Position (§4.1): attention only, experts only, attention plus experts, each with or without a LoRA branch on the router app. C.1.
- Granularity (§4.2): module level (one shared branch, rank 256), expert level (one rank-2 branch per routed expert), and weight level (rank-1 or rank-2 LoRA on each expert's gate, up and down weights, following the native router) app. C.2 fig. 2.
- PEFT method (§4.3): weight-level LoRA, DoRA, OFT, FourierFT and IA3, plus expert-level LoRA, MoE-LoRA, module-level LoRA, DAS-LoRA and last-layer partial fine-tuning app. C.3 tab. 5 tab. 6.
- Auxiliary load-balancing loss (§4.4): a coefficient λ in {0, 1e-4, 1e-3, 1e-2, 1e-1}, with the adapter held fixed at weight-level LoRA rank 1 app. C.4.
The weight-level redesign (§5, app. A.1) changes the MoE2-LoRA baseline in three ways: its routed LoRA pool is placed inside the native expert projections rather than as a module-level bypass after the MoE output; attention stays frozen; and the balancing coefficient is zero. The router keeps selecting pool entries, and its own weights are not updated §5.
Equations for the expert computation and adapter updates are in app. A. The main-text equations are eq. 1 (the MoE output) and eq. 2 (the additive update). Analysis equations are eq. 3 (functional effective rank), eq. 4 (the rank-one response of an up-projection update), eq. 5 (cumulative spectral energy) and eq. 6 (the spectral-consistency score) §6.1 §6.2.
Expert-weight LoRA on one routed expert
# Weight-level LoRA on one routed expert [app. A] [app. C.2]
# Input: x (T, d) hidden states for T tokens routed to this expert
# Frozen: W_gu (2*d_ff, d), W_down (d, d_ff)
# Trainable: A_gu, B_gu, A_dn, B_dn (rank r; r = 1 in the main setting)
def expert_with_lora(x):
# 1. Scale of every low-rank update: alpha / r [app. A] [app. C.2]
scale = alpha / r
# 2. Fused gate and up projection, frozen plus low-rank residual [app. A]
gate_up = x @ (W_gu + scale * B_gu @ A_gu).T # (T, 2*d_ff)
gate, up = split(gate_up)
# 3. Gated hidden state; SiLU is the activation for SwiGLU [app. A]
hidden = silu(gate) * up # (T, d_ff)
# 4. Down projection, frozen plus low-rank residual [app. A]
return hidden @ (W_down + scale * B_dn @ A_dn).TThe router output is then combined over the selected experts as in eq. 1, with the expert outputs as above app. A.
Load-balancing term and total loss
# Per-layer balancing term, as implemented [app. C.4]
# [code: src/llamafactory/model/model_utils/moe_lora.py#L832-L863]
# Input: router_logits (T, E); top_idx (T, k) with k = top_k; E experts
# Output: scalar for one MoE layer
def balance_term(router_logits, top_idx):
# 1. Dense softmax probabilities over all E experts [app. C.4]
probs = softmax(router_logits, axis=-1) # (T, E)
# 2. Hard dispatch counts; each token adds k ones [app. C.4]
# [code: src/llamafactory/model/model_utils/moe_lora.py#L832-L863]
dispatch = one_hot(top_idx, E).sum(axis=1) # (T, E)
f = mean(dispatch, axis=0) # sums to k
# 3. Mean dense probability per expert over the batch [app. C.4]
p = mean(probs, axis=0) # (E,)
# 4. Under uniform routing this equals k [code: run_auxiliary_loss.sh#L8]
return E * sum(f * p)# Total loss with a constant balancing coefficient [app. C.4]
# [code: src/llamafactory/model/model_utils/moe_lora.py#L1737-L1794]
# Input: lm_loss (scalar), layer_terms (one balance_term per MoE layer)
def total_loss(lm_loss, layer_terms, lam):
# 1. Average the per-layer terms over layers [code: moe_lora.py#L1771]
balance = mean(layer_terms)
# 2. Constant coefficient in the sweep; code default decays it [app. C.4]
return lm_loss + lam * balanceDoRA on one expert projection, detached norm
# DoRA on one fused expert projection [app. A]
# [code: src/llamafactory/model/model_utils/weight_peft.py#L872-L879]
# Input: x (T, d_in); frozen W0 (d_out, d_in); scale = alpha / r
# Trainable: A (r, d_in), B (d_out, r), magnitude m (d_out,)
def dora_projection(x):
# 1. Direction matrix V = W0 + scale * B @ A [app. A]
V = W0 + scale * (B @ A)
# 2. Row norms, constant in the backward pass [app. A]
# [code: src/llamafactory/model/model_utils/weight_peft.py#L874-L878]
norms = stop_gradient(row_norm(V)) # (d_out,)
# 3. Trainable magnitude divided by the norm, applied to V [app. A]
return x @ ((m / norms)[:, None] * V).THyperparameters
Values not given in the paper are marked "not stated". Values from the official code are marked as code-only.
| Name | Value | Where | Notes |
|---|---|---|---|
| Backbones | Qwen3-30B-A3B-Base; Gemma-4-26B-A4B; GPT-OSS-20B | app. E tab. 9 tab. 10 | Text pathway only; Gemma-4 vision not used |
| Routed experts / top-k | Qwen3 128 / 8; Gemma-4 128 / 8; GPT-OSS 32 / 4 | tab. 10 | |
| Routed expert width | Qwen3 768; Gemma-4 704; GPT-OSS 2880 | tab. 10 | Gemma-4 also has a frozen always-on shared expert tab. 10 |
| Weight-level LoRA (main) | rank 1, α = 2 (scale 2) | §4.2 tab. 4 | Rank 2, α = 4 is the other arm |
| Router LoRA | rank 8, α = 16 | app. C.1 app. C.2 app. C.3 | Applied in the granularity, method and balancing studies |
| Expert-level LoRA | rank 2, α = 4 per routed expert | tab. 4 | |
| Module-level LoRA | rank 256, α = 512, one top-1 branch per layer | tab. 4 | |
| Attention-only LoRA | rank 64, α = 128 | tab. 3 | Attention+experts uses rank 32 / α = 64 plus expert rank 128 / α = 256 |
| Expert adaptation, position study | rank 256, α = 512 | tab. 3 | Higher-budget arm adds attention rank 64 |
| Learning-rate grid | {1e-5, 1e-4, 1e-3} | app. B.2 | Chosen by mean downstream score over three seeds |
| Selected LR, Qwen3 weight-level LoRA | Intent 1e-3; Math 1e-4 | tab. 14 | Selection detail in app. H |
| Optimizer | AdamW, β1 0.9, β2 0.999, ε 1e-8 | app. B.2 | Paper says "fused AdamW"; the configs in the repository do not set an optimizer, so the trainer default applies (see Paper vs code) |
| Schedule | cosine, warmup ratio 0.1 | app. B.2 | |
| Weight decay, gradient clipping | not stated | app. B.2 | Not set in the paper configs inspected |
| Precision and memory | BF16; gradient checkpointing; DeepSpeed ZeRO-2 | app. B.2 | Four GPUs |
| Effective batch size | 144 (6 per device × 6 accumulation × 4 GPUs) | app. B.2 | Gemma-4 Intent: 4 × 9 × 4 = 144 |
| Adapter dropout | none | app. B.2 | |
| Math training | MetaMathQA-R1; ≤ 80,000 samples; 1 epoch; cutoff 2,048; default packing | tab. 2 | Only the final solution is kept; the think content is removed |
| Intent training | ESFT_Intent; ≤ 80,000 samples; 3 epochs; cutoff 4,096; packing off | tab. 2 | |
| Math evaluation | MATH-500, 500 examples, accuracy; max 512 new tokens; batch 32 | tab. 2 | Decoding not stated in the paper |
| Intent evaluation | 500 held-out examples, exact match; max 512 new tokens; batch 8 | tab. 2 | |
| General evaluation | MMLU-Pro (12,032 examples); IFEval (541 prompts, prompt-level accuracy) | tab. 7 | Same checkpoints as the downstream score |
| Balancing coefficient λ | {0, 1e-4, 1e-3, 1e-2, 1e-1}, constant schedule | app. C.4 | Schedule as used in the sweep; the code default is different (see Paper vs code) |
| FourierFT | scale γ = 150; fixed support, seed 2024; zero-initialised coefficients | app. A | Paper says the seed is fixed; see Paper vs code |
| OFT setting (Qwen3, expert/router) | "4/128" | tab. 5 | The table does not label the two numbers; Gemma-4 "8/128", GPT-OSS "6/32" |
| IA3 | one-initialised vectors; no rank | tab. 5 app. A | |
| MoE-LoRA | independent adapter router, 8 adapters, top-2; rank 32, α = 64 on Qwen3 and Gemma-4 | tab. 6 | Rank 8, α = 16 on GPT-OSS |
| DAS-LoRA | 8 experts, rank 32, α = 64 (Qwen3, Gemma-4); 4 experts, rank 16, α = 32 (GPT-OSS) | tab. 6 | Expert set from the CDAS score app. A |
| Partial fine-tuning | full expert and router weights in the last decoder layer | tab. 6 | |
| LoRA dropout | none | app. B.2 |
Results
Author-reported. Setup: three MoE backbones, two downstream tasks, MMLU-Pro and IFEval retention, three seeds per setting §3.2 app. B. Each figure summarises one design study in the paper's own terms.
Adaptation position §4.1 fig. 3. Averaged over the six backbone-task cells: experts only 68.6%, experts plus router 68.1%, attention plus experts 67.0%, attention only 61.7%. Relative changes on MMLU-Pro and IFEval: experts only −3.3 and −3.6; attention only −10.4 and −7.9; attention plus experts −1.8 and −1.0; experts plus router −2.0 and −3.6. At the higher budget, 64.8% without router adaptation and 64.9% with it §4.1.
Granularity §4.2 fig. 4. Weight-level rank 2: 69.1%. Weight-level rank 1: 68.5%. Module-level bypass (rank 256): 68.4%. Expert-level bypass (rank 2): 64.7%. Rank 1 changes MMLU-Pro by +0.8 and IFEval by −1.4; rank 2 changes them by −0.9 and −3.4. Rank 2 gains 0.6 downstream points over rank 1 at about twice the trainable parameters §4.2.
PEFT method §4.3 fig. 5 tab. 14. LoRA and DoRA, both applied at the weight level, reach the same 68.5% average. DoRA keeps more IFEval (−1.1 vs −1.4 for LoRA). Other results: layer-level LoRA 66.7%, MoE-LoRA 65.7%, DAS-LoRA 64.8%, expert-level LoRA 64.7%, OFT 60.8%, FourierFT 43.6%, IA3 41.3%, last-layer partial fine-tuning 39.4% §4.3.
Auxiliary balancing loss §4.4 fig. 6 app. D.2. Downstream averages for λ = 0, 1e-4, 1e-3, 1e-2, 1e-1: 68.5, 68.1, 66.8, 64.6, 58.6. At λ = 1e-3 the MMLU-Pro change is +1.2, the IFEval change −2.5 and the downstream average −1.7; at λ = 1e-1 the MMLU-Pro and IFEval changes are −0.9 and −8.5 §4.4. The appendix validation on MoE2-LoRA shows λ = 1e-4 raising Intent from 84.00 to 85.00 but lowering MATH-500 from 60.00 to 56.40 tab. 7 tab. 8.
Weight-level redesign of MoE2-LoRA on Qwen3 §5 tab. 1:
| Checkpoint | Benchmark | Original MoE2-LoRA | Weight-level redesign |
|---|---|---|---|
| Trainable parameters | 47.71M | 39.58M | |
| Math | MATH-500 | 60.0 | 55.8 |
| Math | MMLU-Pro | 48.19 | 50.63 |
| Math | IFEval | 35.12 | 40.30 |
| Intent | Intent | 84.0 | 81.8 |
| Intent | MMLU-Pro | 45.26 | 51.20 |
| Intent | IFEval | 18.11 | 36.04 |
The redesign reduces trainable parameters by 17.0%. On Math, the MMLU-Pro gain is 2.44 points and the IFEval gain 5.18; on Intent, the two gains are 5.94 and 17.93. The average absolute deviation from the base model drops from 6.64 to 2.19 points, and the largest loss from 16.27 to 0.87. Downstream cost is 3.2 points on average §5.
Mechanism §6 fig. 7 fig. 8 tab. 11 tab. 12. On Qwen3 (48 layers), weight-level rank 1 reaches a mean functional effective rank of 63.90 on Math and 69.08 on Intent. Expert-level rank 2 gives 16.31 (Math) and 15.91 (Intent). On mean pretrained spectral consistency, weight-level rank 1 scores 0.812 (Math) and 0.733 (Intent), while the expert-level variant scores 0.141 and 0.087. The authors read the spectral result as weight updates reusing the frozen expert's down-projection spectrum §6.2 app. G.
Training cost app. H tab. 14. On Qwen3, weight-level LoRA processes 28.27 samples per second on Intent and 19.97 on Math. DoRA reaches 21.63 and 13.74 and expert-level LoRA 18.47 and 14.12 (author-measured, four GPUs, effective batch 144). The paper reports these as 1.31× and 1.45× DoRA's throughput app. H.
Paper vs code
Repository reviewed: https://github.com/leoli646/MoEFactory at commit
b3916b2a09c0b7609f1c8fd7980f92783e9e56f9, Apache-2.0 [LICENSE]. The paper's
own link points to this repository. The repository contains a copy of a
llamafactory package under src/llamafactory, with the study scripts and
configs added around it.
Differences found:
- Balancing schedule. The paper says the schedule is constant app. C.4. The code default is
linear_decay, which multiplies the coefficient by (1 − progress) finetuning_args.py moe_lora.py moe_lora.py. The auxiliary-loss sweep script setsmoe_lora_balance_loss_schedule=constantrun_auxiliary_loss.sh, which matches the paper. The main protocol YAML leaves the schedule unset and has λ = 0 weight_lora_qwen3_math_lr1e-5_seed42.yaml. - Balancing normalisation. The paper gives the per-layer term E Σ f_e p̄_e without saying how layers are combined or how f_e is counted app. C.4. The code counts every top-k selection (each token contributes k ones), so the term equals k under uniform routing, not 1 moe_lora.py run_auxiliary_loss.sh. The code averages per-layer terms over the layers moe_lora.py.
- Router-LoRA initialisation. The paper gives no initialisation. Router LoRA uses B ~ N(0, 1e-4), not zero moe_lora.py. Expert-weight adapters use zero-initialised B weight_peft.py. The
moe_lora_b_initoption is validated only for bypass arms moe_lora.py. - FourierFT seed. The paper says the spectral support is fixed with seed 2024 app. A. The code takes the base seed 2024 and adds a running counter per adapted weight (
_next_seed), so the first adapter gets seed 2025 weight_peft.py. The γ = 150 scale matches finetuning_args.py. - DoRA norm. The paper gives the full norm and presents the stop-gradient version as the memory-efficient implementation app. A. The code always detaches the norm in the expert path and the generic path weight_peft.py weight_peft.py.
- MoE2 pool size. The paper's reproduced MoE2-LoRA uses two groups of 128 rank-32 entries, 256 in total app. A.1 app. D.1. The example YAML for expert MoE2-LoRA sets 512 entries expert_moe2_lora.yaml. Use the paper's 256 when reproducing the paper's numbers; the example is a different benchmark setting.
- Example configs are not the paper protocol. The
examples/peft_compare/*.yamlconfigs use learning rate 6e-4 and batch 12 × 3 with attention adapters on weight_lora.yaml. They do not match the paper's grid or its no-attention settings. Theexamples/paper/*.yamlconfigs match the paper's batch size, cutoff, router rank and rank-1 weight-level LoRA weight_lora_qwen3_math_lr1e-5_seed42.yaml, but at the single grid point 1e-5. - Learning-rate candidates. The method comparison script uses the paper's grid {1e-5, 1e-4, 1e-3}, with per-model ordering run_method_comparison.sh. This matches.
- Think content. The paper removes the think content from MetaMathQA-R1 responses app. B.1. The data script strips it by default prepare_metamathqa_r1.py.
- Redesign arm. The paper's redesign result (Table 1) corresponds to a combined arm named
n128_r32_wp_up_ntk_a11_bzin the redesign runner run_moe2_redesign_local4.sh. I did not decode its flags, so this card cannot confirm that the arm reproduces Table 1 exactly. - Optimizer. The paper says "fused AdamW" app. B.2. The configs and scripts I read do not set an optimizer, so the framework default applies. Weight decay and gradient clipping are not set anywhere I checked.
- Decoding. The paper does not state a decoding setting. The shared generation helper defaults to greedy decoding (temperature 0) _common.py.
The expert-weight LoRA placement on gate, up and down projections and the α = 2r scaling in the study match the configs. I did not check the router's top-k renormalisation against the code.
Pitfalls
- Balancing coefficient scale. The per-layer term is k under uniform routing in the code, so a coefficient taken from another codebase may mean something different. Check the normalisation before reusing a value.
- Default schedule. A run that sets λ > 0 without the schedule flag gets linear decay, not the constant schedule the paper's sweep used.
- Router init is not zero. The router LoRA starts at a tiny random perturbation, not at the frozen router. The paper does not say this.
- Memory. The frozen 30B-class base must sit in memory during training. The paper's gains are measured with gradient checkpointing and ZeRO-2 on four GPUs; the paper does not report a memory budget.
- Rank 1 vs rank 2. Rank 2 gives more functional diversity, but the paper reports only marginal downstream gain (0.6 points) for about twice the trainable parameters §4.2.
- Attention adaptation. In the appendix validation, adding attention at a matched budget lowers downstream scores on both tasks. The paper advises adapting experts by default app. D.2.
- Balancing is task-dependent. λ = 1e-4 helps Intent but hurts MATH-500 in the appendix runs. The paper's recommendation of "about 1e-4 when routing regularisation is required" is not a task-stable improvement app. D.2.
- Cost of a "safe" default. The paper's most reliable setting, weight-level rank 1, uses rank-1 LoRA on the fused expert weights. Expert weights are stored as fused tensors on these backbones; implementers must handle the fused layout (the code uses a grouped-matmul path with a safe custom backward) weight_peft.py.
- Eval scope. Main-text scores are three-seed means at the selected learning rate, and general-capability scores come from the same checkpoints. The card found no per-seed spread for the accuracy numbers. Decoding is not stated.
- Redesign reproduction. Table 1 depends on the redesign arm configuration, which this card does not decode (see Paper vs code).
Lineage
- Builds on: MoE2-LoRA (Yang et al., 2026; Qingyu Yang, Haonan He, Minglei Li, Jingqi Ye, Tao Chen, Lei Bai, Peng Ye), a routed-pool LoRA method for MoE models. Several authors overlap with this paper. The paper refines it along the one axis it did not follow, the placement of the update inside the expert weights §5 app. A.1. Not carded.
- Compared against: LoRA and DoRA (weight-level and expert-level), OFT, FourierFT, IA3, MoE-LoRA, DAS-LoRA and last-layer partial fine-tuning §4.3 app. C.3. None is carded here.
- Superseded by: not known from content on this wiki.
Source and license
Adapted from “An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs” by Minglei Li, Peng Ye, Haonan He, Jingqi Ye, Qingyu Yang, Tao Chen (arXiv:2610.26370), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/leoli646/MoEFactory at commit b3916b2a09c0 (Apache-2.0). Any quoted code is shown under that license.