Mixture of experts (MoE)

Overview

A mixture-of-experts (MoE) layer holds several expert networks and a router. For each token, the router scores all experts and only the top few run on that token. The parameter count can therefore grow without a matching growth in per-token compute 2106.05974 §1 2112.06905 tab. 4. The layer output is the sum of the selected experts' outputs, each weighted by its router gate 2106.05974 §2.1 2112.06905 §4.

Seven token-routing papers on this page apply the idea to different models: a vision transformer 2106.05974 §1, a decoder-only language model 2112.06905 §4, a one-tower image-and-text contrastive model 2206.02770 §2.2, a decoder-only language model with 8 experts per layer 2401.04088 §2, a multilingual translation model 2207.04672 §6.2, a hybrid Mamba-Transformer language model 2604.12374 §2.1, and a multimodal language model with LoRA experts in its feed-forward layers 2602.01990 §3. They differ in where the experts sit, how many experts a token uses, how overflow is handled, and how load is balanced. One more paper uses a MoE base in a video diffusion model, and it picks experts by denoising noise level rather than by token 2511.23429 §4.1. A navigation agent is another case. Its router reads the agent's current state, not a token 2412.05552 §3.2.

The Gemini 1.5 report describes Gemini 1.5 Pro as a sparse MoE Transformer. A learned routing function sends each input to a subset of the parameters 2403.05530 §3.1 app. model card. The report gives no expert count, top-k value or routing function 2403.05530 §3.1. So it is evidence that the design exists, but not enough to implement it, and it adds no variant below.

Mixture of Horizons 2511.19433 §3.2 is listed under this concept, but it is not an expert-routing method. Its gate is a softmax over predictions for several action-chunk lengths produced by one shared action transformer. The paper only borrows the MoE load-balancing idea for its balance loss 2511.19433 §3.2. It is not used as evidence for any variant below.

A motion VAE is a dense case. Its decoder holds six expert networks and a gating network that blends their outputs. The paper describes a weighted blend, not a top-k selection, so no expert is skipped for a given frame 2103.14274 §3.2. It is covered in its own variant below.

The token-routing evidence comes from seven papers, each reporting its own setup. The sparse-MoE layer that V-MoE and GLaM build on, from Shazeer et al. (2017), has no card on wikillm, so nothing from that work is used here.

Variants

Sparse top-k MoE in a ViT (V-MoE)

Each patch token goes to its top k = 2 of E = 32 experts. The router is a linear map of the patch representation, followed by a softmax and then a top-k selection. During training, noise with standard deviation 1/E is added to the router logits 2106.05974 §2.3. Each expert has a fixed buffer per batch, set by a capacity ratio of 1.05 upstream. Tokens that arrive after the buffer is full skip that expert and rely on the residual path 2106.05974 §2.4 §3.1.

The paper reports V-MoE-H/14 with Every-2 placement (7.2B parameters) at 60.62 JFT precision@1 upstream, against 56.68 for the dense ViT-H/14 2106.05974 tab. 2. The same table prints more training compute for the MoE model (5.79k ExaFLOPs) than for ViT-H/14 (4.27k) 2106.05974 tab. 2, so the gain is not a training-compute saving in that comparison. Card: Scaling Vision with Sparse Mixture of Experts.

Expert placement: every second block or the last n blocks

Every-2 puts an MoE layer in every other block, following GShard. Last-n puts them only in the last n even-numbered blocks, with n = 5 for H/14 and n = 2 otherwise 2106.05974 §3.1 §3.3. The authors state that Last-n has less communication overhead, so it runs faster, and that it has little effect on quality in their setup 2106.05974 §3.1 app. E.

In the table, V-MoE-H/14 Last-5 (2.7B parameters) reaches 60.12 JFT and 88.23 ImageNet fine-tuned. Every-2 (7.2B) reaches 60.62 and 88.36. Training compute is 4.75k and 5.79k ExaFLOPs 2106.05974 tab. 2.

Batch prioritized routing at inference (BPR)

BPR changes only the allocation rule, not the architecture. Each token gets a priority equal to its largest router weight. Tokens are sorted across the whole batch, and buffers are filled choice by choice, so every top-1 choice is placed before any top-2 choice 2106.05974 §4.1 app. C.1. Vanilla routing instead visits tokens in a fixed row order 2106.05974 §4.1.

BPR can be applied to a model trained with vanilla routing, with its weights unchanged. The capacity ratio C at inference sets the compute budget 2106.05974 §4.2 app. C.2. In the FLOPs-matched table, V-MoE-H/14 with BPR at C = 0.5 and k = 2 reaches 58.61 JFT (Last-5) and 59.44 JFT (Every-2), against 56.68 for dense ViT-H/14. Both rows run at about 100% of dense FLOPs, with time at 118% and 156% of dense 2106.05974 app. C. The paper's text makes a stronger efficiency claim for this method, which the tables do not show. See Contradictions.

Two costs follow from the design. First, the token ranking is global across the batch, so one image's output can depend on the other images in its batch 2106.05974 §4.1. Second, very low capacity hurts accuracy. For a V-MoE-S/32 model fine-tuned with k = 2, accuracy falls from 77.4% to 68.6% when C drops to 0.1 2106.05974 app. C.4.

One-tower image-text MoE with entropy routing losses (LIMoE)

LIMoE puts images and captions through one shared transformer. Each MoE layer has E = 32 experts, and each token goes to its top K = 1 expert, with the paper's default setting 2206.02770 §2.2 app. A §3.1. The router is one dense layer shared by both modalities 2206.02770 app. E. The paper reports that mixing modalities in one router makes a single expert take all the minority modality's tokens and drop them once full. The V-MoE balance losses do not stop this, because the total load can stay balanced while one modality is dropped 2206.02770 §2.2.1. The fix has two parts. Per-token (local) and per-modality (global) entropy losses are added to the classic losses 2206.02770 §2.2.2. Overflow is settled by batch priority routing, ordered by the largest router weight 2206.02770 §2.2.3. The auxiliary losses are combined with one weight of 0.04, and the paper does not fully state how they are combined 2206.02770 app. B. The final combination keeps the global entropy loss on both modalities and the local entropy loss on text only. The local loss on images was harmful in the sweep, and the global image loss is almost never active 2206.02770 §4.1 [app. B].

Against dense one-tower baselines at matched compute, the paper reports average zero-shot gains of 10.1 points and 10-shot gains of 12.2 points across scales 2206.02770 §3.1. In the scaling study, zero-shot ImageNet is 73.7 for LIMoE B/16 and 64.3 for dense B/16 2206.02770 app. C. In the shorter 100k-step ablation, the gap is smaller: 56.9 for LIMoE against 49.8 for the dense one-tower model 2206.02770 §4. The largest model, LIMoE-H/14, reports 84.1 zero-shot. It is trained with JFT-4B co-training and a model soup of eight checkpoints, and the soup is selected on ImageNet validation accuracy 2206.02770 §3.2 app. A.3. Treat that number as an optimistic comparison point. The compute claims also depend on the measure. The paper calls LIMoE Pareto-optimal in FLOPs, but its COCO retrieval step-time gain over dense models is not significant 2206.02770 app. D. Card: Multimodal Contrastive Learning with LIMoE.

Its batch priority routing is applied during training. It has the same name as the inference-time BPR above, but the two papers' uses differ. Unimodal evaluation needs a higher capacity factor (16) to avoid heavy token dropping 2206.02770 app. D.

Top-2 MoE in a decoder-only language model (GLaM)

GLaM replaces the feed-forward sub-layer of every other Transformer block with an MoE layer of 64 experts, and sends each token to its top two experts 2112.06905 §4 tab. 4. The 64B/64E model has 1.2T total parameters and 96.6B active per token 2112.06905 tab. 4. The paper compares it with dense GPT-3 (175B) on zero-, one- and few-shot averages, where it scores 62.7, 65.5 and 68.1 against 56.9, 61.6 and 65.2 2112.06905 tab. 1. Its inference FLOPs per token are 180G against 350G 2112.06905 tab. 1. All of these are author-reported.

Two scaling results are only plotted, not tabled. At similar effective FLOPs per token, MoE models score better than dense ones on generation tasks, and at larger scales on understanding tasks 2112.06905 §6.3 fig. 3. At a fixed compute budget per token, average scores rise as the expert count grows from 1 to 256 on the 1.7B base 2112.06905 app. B fig. 6. Card: GLaM: Efficient Scaling of Language Models with Mixture-of-Experts.

Latent-space MoE in a hybrid Mamba-Transformer (LatentMoE)

Nemotron 3 Super pairs Mamba-2 blocks with MoE layers and keeps a few attention layers 2604.12374 §2.1.3. Each MoE layer has 512 experts, and the router picks the top 22 per token. The paper names a sigmoid score with an expert bias for this router 2604.12374 tab. 1 §2.4.

LatentMoE changes where routing happens. Each token is projected from width d = 4096 to a latent width of 1024, the routed experts run in that latent space, and the output is projected back to d. The shared expert and the gate stay at full width 2604.12374 §2.1.1 tab. 1. The authors argue that this cuts routed-expert traffic and weight loads by d/l, and that the saving pays for N′ = N·d/l total experts and K′ = K·d/l active experts at similar cost 2604.12374 §2.1.1. Here d/l is 4 (our arithmetic). The card does not state the baseline N and K.

Load balance uses an aux-loss-free expert-bias update at rate 1e-3, plus a load-balancing loss at 1e-4 2604.12374 §2.4. Throughput is reported as 2.2x and 7.5x that of GPT-OSS-120B and Qwen3.5-122B at 8k input and 64k output, but the competitors run in different precisions 2604.12374 §1 fig. 1. Card: Nemotron 3 Super.

LoRA experts with spectral-aware routing for continual tuning (SAME)

SAME puts a mixture of LoRA experts and a router in the feed-forward layers of the language model in LLaVA-v1.5-7B. The vision encoder and projector stay frozen 2602.01990 §3 §5.1. The model learns tasks one after another without replayed data 2602.01990 §1 §5.1. Three changes limit forgetting. The router is updated mostly in the input directions that the new task uses. Expert updates are preconditioned by the previous tasks' input covariance. Experts that the current task barely uses are frozen for that task only 2602.01990 §4.1 §4.2 §4.3.

The authors report average accuracy of 66.82 on CoIN and 67.12 on UCIT, against 50.58 and 52.06 for MoELoRA 2602.01990 tab. 2 tab. 3. On CoIN, the ablation adds the three changes in turn: 50.58 for the baseline, 61.32 with the router change, 65.89 with the expert change and 66.82 with the freezing rule 2602.01990 tab. 4. On the paper's ten-task TriGap sequence, the average is 46.53 against 44.45 for MoE-LoRA. SAME is below MoE-LoRA on CLEVR (54.50 against 67.00) 2602.01990 tab. 1. All of these are single-run numbers, with no seeds or error bars reported 2602.01990 §5.1. The released code differs from the paper's equations in several places, so the card's Paper vs code section should be read before any of these numbers are reproduced. Card: SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning.

State-adaptive experts on a navigation agent's visual query (SAME, navigation)

This SAME is a different method from the continual-tuning SAME above. It puts a sparse MoE layer on the visual query of the cross-attention in DUET, a dual-scale graph transformer for language-guided navigation 2412.05552 §3.4 §3.1. The router takes the mean of the panorama view features, merged with the instruction [CLS] token through one linear layer, and picks the top-k experts for each step 2412.05552 §3.2. The paper gives the routing formula but states neither the number of experts nor k 2412.05552 §3.1.

In single-run ablations, the state-adaptive router reaches 45.67 success on REVERIE, against 43.45 for a token-embedding router and 43.71 for a task-embedding router 2412.05552 tab. 2. Among the three expert placements tested, the visual query gives the best SPL on all three ablation tasks 2412.05552 tab. 3.

The model is trained once on seven navigation datasets with DAgger. The balance term is weighted 0.8 2412.05552 §4. Against the authors' multi-task run of ScaleVLN, it matches on R2R validation success (76) and scores higher on REVERIE (46.4 against 41.9). ScaleVLN trained on a single task is higher on both (79 on R2R, 57.0 on REVERIE) 2412.05552 tab. 4. In the single-run ablations, batches that mix datasets cut R2R success from 73.69 to 62.36, so the paper keeps one dataset per batch [arxiv:2412.05552 tab. 6] §4. Without ScaleVLN pre-training, R2R success falls to 59.05 2412.05552 tab. 2.

All of these are author-reported, and tables 2, 3, 6 and 7 are single runs. The released code differs from the paper in several places, so the card's Paper vs code section should be read before any number is reproduced. Card: SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts.

Top-2 of 8 SwiGLU experts in every layer of a decoder-only model (Mixtral)

Mixtral 8x7B replaces the feed-forward block of each layer of a Mistral 7B-style Transformer with a sparse layer of 8 SwiGLU experts 2401.04088 §2 tab. 1. A linear router without bias picks 2 experts per token. The softmax is taken over those two logits only, so their weights sum to 1 2401.04088 §2.1. The paper reports about 47B stored parameters, about 13B used per token, and a dense 32k context 2401.04088 §3 tab. 2. The text describes no capacity limit, no token dropping and no load-balancing loss 2401.04088 §2.1.

The paper's caption says Mixtral matches or beats Llama 2 70B on almost all of its benchmarks. Llama 2 70B is ahead on HellaSwag, WinoGrande and TriviaQA 2401.04088 tab. 2. On MMLU the scores are 70.6 for Mixtral against 69.9 for Llama 2 70B 2401.04088 tab. 2. The routing analysis finds no clear topic specialisation of experts 2401.04088 §5 fig. 7. Card: Mixtral of Experts.

Top-2 of 128 experts with expert output masking in a translation model (NLLB-200)

NLLB-200 replaces the feed-forward sub-layer of every fourth Transformer block with a sparse layer of 128 experts, and each token goes to its top two 2207.04672 §8.2.2 §6.2. The paper gives 54.5B total parameters, with FLOPs similar to a 3.3B dense model 2207.04672 §8.2.4. The gate is a softmax over a linear map of the token 2207.04672 §6.2 eq. 7.

The training loss adds a load-balancing term with weight 0.01 2207.04672 §6.2 eq. 9. Each expert has a training capacity of 2 × ceil(T / E) for T tokens in the mini-batch and E experts. At generation, the capacity is set to the whole batch 2207.04672 §8.2.4.

Expert output masking (EOM) is the paper's own regulariser. For a random fraction p of tokens, the first and/or second expert's output is zeroed during training. The final model uses p = 0.2 2207.04672 §6.2.2 §8.2.4. In the MoE-64 ablation on the dev set, EOM at p = 0.1 scores 44.7 chrF++ on the English-to-other pairs, against 44.3 for dropout at its best setting 2207.04672 tab. 16. Conditional MoE routing scores 44.8 in that table, slightly higher, but the authors state it costs 18 to 25% more training time, so they chose EOM 2207.04672 §6.2.2 tab. 16. The ablations use a peak learning rate of 0.004, against 0.002 for the final model 2207.04672 §6.2.1 §8.2.4. Card: No Language Left Behind: Scaling Human-Centered Machine Translation.

Dense gated blend of six experts in a motion VAE decoder (MVAE)

This variant does not route tokens. The decoder of a motion VAE predicts the next pose of a character from the previous pose and a 32-dimensional latent 2103.14274 §3.1 §3.2. The decoder has six experts, each with three layers of 256 ELU units. A gating network with 256 hidden units blends them 2103.14274 §3.2. The latent is fed into every layer of every expert, which the authors say limits posterior collapse 2103.14274 §3.2 §3.3. The gate takes the latent and the previous pose, and the latent is used only in its first layer 2103.14274 §3.2.

The expert count is ablated. The tables compare six and four experts, both at beta = 0.2 2103.14274 tab. 1. Foot skate with six experts is lower than with four in four of the five columns: 0.067 against 0.085 on random walks, 0.28 against 0.39 on joystick, 0.30 against 0.44 on path following and 0.24 against 0.38 in the maze. The target column is the exception, at 0.27 against 0.24 2103.14274 tab. 1. Four experts reach a target direction faster in the joystick test, at 1.38 seconds against 1.62 2103.14274 tab. 2. These are single runs with no variance reported 2103.14274 tab. 1. The released code uses a gating width of 64, not the paper's 256, so the card's Paper vs code section should be read before the gate is implemented. Card: Character Controllers Using Motion VAEs.

Noise-level experts in a video diffusion model

This variant is not a token router. The base is a 14B image-to-video MoE model, cited as Wan, with two experts: one for the high-noise steps and one for the low-noise steps 2511.23429 §4.1. The distillation timesteps are redefined around the noise boundary that separates the two experts, and each expert gets its own learning rate. The values are not stated 2511.23429 §4.2.3. In supervised finetuning only the MoE experts are trained, and the camera encoder is frozen 2511.23429 §4.2.2. Card: Hunyuan-GameCraft-2.

The paper's only ablation on the card covers cache and long-video settings 2511.23429 fig. 16. It does not isolate the effect of the expert split, so this variant has no measured benefit on this page.

When to use which

Implementation sketch

The token-routing variants share one loop, with differences in the gate and in the overflow rule. V-MoE uses a softmax gate, fixed buffers, a visiting order and a capacity ratio C 2106.05974 §2.3 §4.1 §2.4. GLaM uses a softmax over experts, but its cards state no capacity limit 2112.06905 §4. Nemotron uses a sigmoid score with an expert bias 2604.12374 §2.4 and states no capacity limit either. SAME uses a softmax gate over its LoRA experts and weights their outputs by the gate 2602.01990 eq. 3. It states no capacity limit, and the top-k cut appears only in its inference algorithm, without a value for k 2602.01990 alg. 2. Mixtral takes its softmax over the two kept logits only, so the weights sum to 1 2401.04088 §2.1. NLLB takes a softmax over all experts and keeps the top two 2207.04672 §6.2 eq. 7, and its training capacity is 2 × ceil(T / E) 2207.04672 §8.2.4. LIMoE uses the same top-k gate and capacity rule, but it adds routing losses to the training objective and applies its priority rule during training 2206.02770 §2.2 app. B. The noise-level variant has no router loop. Its expert is picked by the denoising step, and the card gives no code for that rule 2511.23429 §4.2.3. The navigation variant keeps the same gate form but routes once per sample per step, from a pooled view feature merged with the instruction token, and adds the selected experts' outputs to the visual query 2412.05552 §3.1 §3.2 §3.4. The motion VAE decoder is a third loop. It blends its six expert outputs with gate weights, and the paper describes no top-k cut, so every expert runs on each frame 2103.14274 §3.2.

# Sparse MoE layer over patch tokens  [§2.1] [§2.3] [§2.4]
# x: (N*P, D) tokens; E experts; k selected per token; C capacity ratio
def moe_layer(x, W_router, experts, k, C, train):
    # Router logits, one per expert  [§2.3]
    logits = x @ W_router                              # (N*P, E)
    # Training-only noise, std 1/E; test-time noise not stated  [§2.3]
    if train:
        logits = logits + normal(0, 1 / E)
    # Softmax first, then keep the top-k experts  [§2.3]
    gates = softmax(logits)                            # (N*P, E)
    # Buffer size per expert  [§2.4]
    cap = round(k * N * P * C / E)
    # Renormalising the top-k gates: not stated  [§2.3]
    fill = zeros(E)
    out = zeros_like(x)
    # Visiting order decides overflow: vanilla = row order  [§4.1]
    # BPR = sorted by priority across the batch  [§4.1]
    for t in visit_order(gates):
        for e in top_k_indices(gates[t], k):
            if fill[e] < cap:
                fill[e] += 1
                # Gate-weighted sum of selected experts' outputs  [§2.1]
                out[t] += gates[t, e] * experts[e](x[t])
            # Overflow: this expert skips the token  [§2.4]
            # The residual path carries the token forward  [§2.4]
    return out

Contradictions

• BPR efficiency: text against tables. The text says some models beat dense ViT-H while using under half the FLOPs and under 60% of the runtime 2106.05974 §4.2. The unmatched table it cites is built with vanilla routing, not BPR. In that table, H/14 Last-5 runs at 111.12% of dense FLOPs and 120.22% of dense time, and Every-2 at 135.59% and 164.89% 2106.05974 app. C. The FLOPs-matched BPR table lists the models that beat dense at about 100% FLOPs, with time at 118% to 156% [arxiv:2106.05974 app. C]. The paper also points to a low-capacity figure [arxiv:2106.05974 §4.2] fig. 5. This page did not check that figure.
• Upstream number for V-MoE-L/16 (Every-2). The main table gives 57.65 JFT precision@1 2106.05974 tab. 2. The appendix table, with vanilla routing, gives 57.64 with the same ImageNet 5-shot score of 77.10 2106.05974 app. C. The paper does not explain the difference.
• SAME's router: paper against released code. The paper's layer output sums each expert's gated update, computed per input 2602.01990 eq. 3. The released code routes once per batch, mixes the expert factors in a different way and uses a different expert rank SAME card. The page reports the paper's equations, not the code's behaviour.
• SAME (navigation): text against its own table. The text says lambda = 0.2 "performs worse than all other variants" 2412.05552 §4.2. In table 7, lambda = 0.2 gives the best R2R SPL and is near the best on ObjectNav SPL, but it has the lowest REVERIE success 2412.05552 tab. 7. The claim holds for some metrics only. The released code also differs from the paper's softmax axis and balance-loss scale SAME navigation card, so the two sources are not interchangeable.
• Compute baselines differ. GLaM reports lower inference FLOPs and training energy than GPT-3 2112.06905 tab. 1. V-MoE's table shows more training compute than the dense ViT-H/14 2106.05974 tab. 2. The two papers compare against different dense models and in different domains, so their compute claims are not directly comparable and are not reconciled here.

Papers

• Scaling Vision with Sparse Mixture of Experts: introduces V-MoE, top-k routing with capacity limits in a ViT, and Batch Prioritized Routing for inference.
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts: a decoder-only language model with 64 experts per MoE layer and top-2 routing, compared with GPT-3 on zero-, one- and few-shot tasks.
• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning: a hybrid Mamba-Transformer with 512 experts and top-22 routing in a latent space, pretrained with NVFP4 GEMMs.
• Mixture of Horizons in Action Chunking: listed under this concept, but its gate mixes horizon predictions rather than experts. See the Overview.
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning: mixture of LoRA experts in the feed-forward layers of a multimodal LLM, with constrained router and expert updates for sequential tasks without replay.
• SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts: a mixture-of-experts layer on the visual query of the DUET navigation agent, routed by the agent's state, trained once on seven navigation datasets.
• Character Controllers Using Motion VAEs: a motion VAE whose decoder blends six experts with a gate, with an RL policy that steers its latent.
• Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model: uses a 14B MoE video base with high- and low-noise experts, inside a larger distilled game-world system.
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context: a closed report that calls Gemini 1.5 Pro a sparse MoE Transformer, with no expert count or routing details.
• Multimodal Contrastive Learning with LIMoE: one shared image-text transformer with top-1 routing over 32 experts, per-modality entropy losses and batch priority routing during training.
• Mixtral of Experts: a decoder-only language model whose layers each hold 8 SwiGLU experts, with a top-2 router and open weights.
• No Language Left Behind: Scaling Human-Centered Machine Translation: a 54.5B translation model with 128 experts in every fourth block, top-2 routing, and expert output masking.