--- kind: concept slug: mixture-of-experts title: Mixture of experts (MoE) summary: >- A router sends each token to a few expert sub-networks, so a layer holds more parameters than it runs per token. Variants differ in expert placement, routing losses and overflow handling. A motion VAE blends six experts with a gate. Results are author-reported. A video model picks experts by noise level; one closed model report names MoE without numbers. tags: [mixture-of-experts, sparsity, routing, conditional-computation, inference, language-models, diffusion, multimodal, motion-synthesis] papers: ["2103.14274", "2106.05974", "2112.06905", "2206.02770", "2207.04672", "2401.04088", "2403.05530", "2412.05552", "2511.19433", "2511.23429", "2602.01990", "2604.12374"] related: [vision-transformer, efficient-vision-backbone, self-attention, contrastive-learning, vision-language-pretraining, diffusion-models, language-model-pretraining, state-space-model, low-precision-arithmetic, continual-learning, low-rank-adaptation, multimodal-language-model, neural-machine-translation, imitation-learning, topological-map-navigation] status: draft updated_at: 2026-11-01 --- ## Overview A mixture-of-experts (MoE) layer holds several expert networks and a router. For each token, the router scores all experts and only the top few run on that token. The parameter count can therefore grow without a matching growth in per-token compute [arxiv:2106.05974 §1] [arxiv:2112.06905 tab. 4]. The layer output is the sum of the selected experts' outputs, each weighted by its router gate [arxiv:2106.05974 §2.1] [arxiv:2112.06905 §4]. Seven token-routing papers on this page apply the idea to different models: a vision transformer [arxiv:2106.05974 §1], a decoder-only language model [arxiv:2112.06905 §4], a one-tower image-and-text contrastive model [arxiv:2206.02770 §2.2], a decoder-only language model with 8 experts per layer [arxiv:2401.04088 §2], a multilingual translation model [arxiv:2207.04672 §6.2], a hybrid Mamba-Transformer language model [arxiv:2604.12374 §2.1], and a multimodal language model with LoRA experts in its feed-forward layers [arxiv:2602.01990 §3]. They differ in where the experts sit, how many experts a token uses, how overflow is handled, and how load is balanced. One more paper uses a MoE base in a video diffusion model, and it picks experts by denoising noise level rather than by token [arxiv:2511.23429 §4.1]. A navigation agent is another case. Its router reads the agent's current state, not a token [arxiv:2412.05552 §3.2]. The Gemini 1.5 report describes Gemini 1.5 Pro as a sparse MoE Transformer. A learned routing function sends each input to a subset of the parameters [arxiv:2403.05530 §3.1] [app. model card]. The report gives no expert count, top-k value or routing function [arxiv:2403.05530 §3.1]. So it is evidence that the design exists, but not enough to implement it, and it adds no variant below. Mixture of Horizons [arxiv:2511.19433 §3.2] is listed under this concept, but it is not an expert-routing method. Its gate is a softmax over predictions for several action-chunk lengths produced by one shared action transformer. The paper only borrows the MoE load-balancing idea for its balance loss [arxiv:2511.19433 §3.2]. It is not used as evidence for any variant below. A motion VAE is a dense case. Its decoder holds six expert networks and a gating network that blends their outputs. The paper describes a weighted blend, not a top-k selection, so no expert is skipped for a given frame [arxiv:2103.14274 §3.2]. It is covered in its own variant below. The token-routing evidence comes from seven papers, each reporting its own setup. The sparse-MoE layer that V-MoE and GLaM build on, from Shazeer et al. (2017), has no card on wikillm, so nothing from that work is used here. ## Variants ### Sparse top-k MoE in a ViT (V-MoE) Each patch token goes to its top k = 2 of E = 32 experts. The router is a linear map of the patch representation, followed by a softmax and then a top-k selection. During training, noise with standard deviation 1/E is added to the router logits [arxiv:2106.05974 §2.3]. Each expert has a fixed buffer per batch, set by a capacity ratio of 1.05 upstream. Tokens that arrive after the buffer is full skip that expert and rely on the residual path [arxiv:2106.05974 §2.4] [§3.1]. The paper reports V-MoE-H/14 with Every-2 placement (7.2B parameters) at 60.62 JFT precision@1 upstream, against 56.68 for the dense ViT-H/14 [arxiv:2106.05974 tab. 2]. The same table prints more training compute for the MoE model (5.79k ExaFLOPs) than for ViT-H/14 (4.27k) [arxiv:2106.05974 tab. 2], so the gain is not a training-compute saving in that comparison. Card: [Scaling Vision with Sparse Mixture of Experts](/papers/2106.05974). ### Expert placement: every second block or the last n blocks Every-2 puts an MoE layer in every other block, following GShard. Last-n puts them only in the last n even-numbered blocks, with n = 5 for H/14 and n = 2 otherwise [arxiv:2106.05974 §3.1] [§3.3]. The authors state that Last-n has less communication overhead, so it runs faster, and that it has little effect on quality in their setup [arxiv:2106.05974 §3.1] [app. E]. In the table, V-MoE-H/14 Last-5 (2.7B parameters) reaches 60.12 JFT and 88.23 ImageNet fine-tuned. Every-2 (7.2B) reaches 60.62 and 88.36. Training compute is 4.75k and 5.79k ExaFLOPs [arxiv:2106.05974 tab. 2]. ### Batch prioritized routing at inference (BPR) BPR changes only the allocation rule, not the architecture. Each token gets a priority equal to its largest router weight. Tokens are sorted across the whole batch, and buffers are filled choice by choice, so every top-1 choice is placed before any top-2 choice [arxiv:2106.05974 §4.1] [app. C.1]. Vanilla routing instead visits tokens in a fixed row order [arxiv:2106.05974 §4.1]. BPR can be applied to a model trained with vanilla routing, with its weights unchanged. The capacity ratio C at inference sets the compute budget [arxiv:2106.05974 §4.2] [app. C.2]. In the FLOPs-matched table, V-MoE-H/14 with BPR at C = 0.5 and k = 2 reaches 58.61 JFT (Last-5) and 59.44 JFT (Every-2), against 56.68 for dense ViT-H/14. Both rows run at about 100% of dense FLOPs, with time at 118% and 156% of dense [arxiv:2106.05974 app. C]. The paper's text makes a stronger efficiency claim for this method, which the tables do not show. See Contradictions. Two costs follow from the design. First, the token ranking is global across the batch, so one image's output can depend on the other images in its batch [arxiv:2106.05974 §4.1]. Second, very low capacity hurts accuracy. For a V-MoE-S/32 model fine-tuned with k = 2, accuracy falls from 77.4% to 68.6% when C drops to 0.1 [arxiv:2106.05974 app. C.4]. ### One-tower image-text MoE with entropy routing losses (LIMoE) LIMoE puts images and captions through one shared transformer. Each MoE layer has E = 32 experts, and each token goes to its top K = 1 expert, with the paper's default setting [arxiv:2206.02770 §2.2] [app. A] [§3.1]. The router is one dense layer shared by both modalities [arxiv:2206.02770 app. E]. The paper reports that mixing modalities in one router makes a single expert take all the minority modality's tokens and drop them once full. The V-MoE balance losses do not stop this, because the total load can stay balanced while one modality is dropped [arxiv:2206.02770 §2.2.1]. The fix has two parts. Per-token (local) and per-modality (global) entropy losses are added to the classic losses [arxiv:2206.02770 §2.2.2]. Overflow is settled by batch priority routing, ordered by the largest router weight [arxiv:2206.02770 §2.2.3]. The auxiliary losses are combined with one weight of 0.04, and the paper does not fully state how they are combined [arxiv:2206.02770 app. B]. The final combination keeps the global entropy loss on both modalities and the local entropy loss on text only. The local loss on images was harmful in the sweep, and the global image loss is almost never active [arxiv:2206.02770 §4.1] [app. B]. Against dense one-tower baselines at matched compute, the paper reports average zero-shot gains of 10.1 points and 10-shot gains of 12.2 points across scales [arxiv:2206.02770 §3.1]. In the scaling study, zero-shot ImageNet is 73.7 for LIMoE B/16 and 64.3 for dense B/16 [arxiv:2206.02770 app. C]. In the shorter 100k-step ablation, the gap is smaller: 56.9 for LIMoE against 49.8 for the dense one-tower model [arxiv:2206.02770 §4]. The largest model, LIMoE-H/14, reports 84.1 zero-shot. It is trained with JFT-4B co-training and a model soup of eight checkpoints, and the soup is selected on ImageNet validation accuracy [arxiv:2206.02770 §3.2] [app. A.3]. Treat that number as an optimistic comparison point. The compute claims also depend on the measure. The paper calls LIMoE Pareto-optimal in FLOPs, but its COCO retrieval step-time gain over dense models is not significant [arxiv:2206.02770 app. D]. Card: [Multimodal Contrastive Learning with LIMoE](/papers/2206.02770). Its batch priority routing is applied during training. It has the same name as the inference-time BPR above, but the two papers' uses differ. Unimodal evaluation needs a higher capacity factor (16) to avoid heavy token dropping [arxiv:2206.02770 app. D]. ### Top-2 MoE in a decoder-only language model (GLaM) GLaM replaces the feed-forward sub-layer of every other Transformer block with an MoE layer of 64 experts, and sends each token to its top two experts [arxiv:2112.06905 §4] [tab. 4]. The 64B/64E model has 1.2T total parameters and 96.6B active per token [arxiv:2112.06905 tab. 4]. The paper compares it with dense GPT-3 (175B) on zero-, one- and few-shot averages, where it scores 62.7, 65.5 and 68.1 against 56.9, 61.6 and 65.2 [arxiv:2112.06905 tab. 1]. Its inference FLOPs per token are 180G against 350G [arxiv:2112.06905 tab. 1]. All of these are author-reported. Two scaling results are only plotted, not tabled. At similar effective FLOPs per token, MoE models score better than dense ones on generation tasks, and at larger scales on understanding tasks [arxiv:2112.06905 §6.3] [fig. 3]. At a fixed compute budget per token, average scores rise as the expert count grows from 1 to 256 on the 1.7B base [arxiv:2112.06905 app. B] [fig. 6]. Card: [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](/papers/2112.06905). ### Latent-space MoE in a hybrid Mamba-Transformer (LatentMoE) Nemotron 3 Super pairs Mamba-2 blocks with MoE layers and keeps a few attention layers [arxiv:2604.12374 §2.1.3]. Each MoE layer has 512 experts, and the router picks the top 22 per token. The paper names a sigmoid score with an expert bias for this router [arxiv:2604.12374 tab. 1] [§2.4]. LatentMoE changes where routing happens. Each token is projected from width d = 4096 to a latent width of 1024, the routed experts run in that latent space, and the output is projected back to d. The shared expert and the gate stay at full width [arxiv:2604.12374 §2.1.1] [tab. 1]. The authors argue that this cuts routed-expert traffic and weight loads by d/l, and that the saving pays for N′ = N·d/l total experts and K′ = K·d/l active experts at similar cost [arxiv:2604.12374 §2.1.1]. Here d/l is 4 (our arithmetic). The card does not state the baseline N and K. Load balance uses an aux-loss-free expert-bias update at rate 1e-3, plus a load-balancing loss at 1e-4 [arxiv:2604.12374 §2.4]. Throughput is reported as 2.2x and 7.5x that of GPT-OSS-120B and Qwen3.5-122B at 8k input and 64k output, but the competitors run in different precisions [arxiv:2604.12374 §1] [fig. 1]. Card: [Nemotron 3 Super](/papers/2604.12374). ### LoRA experts with spectral-aware routing for continual tuning (SAME) SAME puts a mixture of LoRA experts and a router in the feed-forward layers of the language model in LLaVA-v1.5-7B. The vision encoder and projector stay frozen [arxiv:2602.01990 §3] [§5.1]. The model learns tasks one after another without replayed data [arxiv:2602.01990 §1] [§5.1]. Three changes limit forgetting. The router is updated mostly in the input directions that the new task uses. Expert updates are preconditioned by the previous tasks' input covariance. Experts that the current task barely uses are frozen for that task only [arxiv:2602.01990 §4.1] [§4.2] [§4.3]. The authors report average accuracy of 66.82 on CoIN and 67.12 on UCIT, against 50.58 and 52.06 for MoELoRA [arxiv:2602.01990 tab. 2] [tab. 3]. On CoIN, the ablation adds the three changes in turn: 50.58 for the baseline, 61.32 with the router change, 65.89 with the expert change and 66.82 with the freezing rule [arxiv:2602.01990 tab. 4]. On the paper's ten-task TriGap sequence, the average is 46.53 against 44.45 for MoE-LoRA. SAME is below MoE-LoRA on CLEVR (54.50 against 67.00) [arxiv:2602.01990 tab. 1]. All of these are single-run numbers, with no seeds or error bars reported [arxiv:2602.01990 §5.1]. The released code differs from the paper's equations in several places, so the card's Paper vs code section should be read before any of these numbers are reproduced. Card: [SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning](/papers/2602.01990). ### State-adaptive experts on a navigation agent's visual query (SAME, navigation) This SAME is a different method from the continual-tuning SAME above. It puts a sparse MoE layer on the visual query of the cross-attention in DUET, a dual-scale graph transformer for language-guided navigation [arxiv:2412.05552 §3.4] [§3.1]. The router takes the mean of the panorama view features, merged with the instruction [CLS] token through one linear layer, and picks the top-k experts for each step [arxiv:2412.05552 §3.2]. The paper gives the routing formula but states neither the number of experts nor k [arxiv:2412.05552 §3.1]. In single-run ablations, the state-adaptive router reaches 45.67 success on REVERIE, against 43.45 for a token-embedding router and 43.71 for a task-embedding router [arxiv:2412.05552 tab. 2]. Among the three expert placements tested, the visual query gives the best SPL on all three ablation tasks [arxiv:2412.05552 tab. 3]. The model is trained once on seven navigation datasets with DAgger. The balance term is weighted 0.8 [arxiv:2412.05552 §4]. Against the authors' multi-task run of ScaleVLN, it matches on R2R validation success (76) and scores higher on REVERIE (46.4 against 41.9). ScaleVLN trained on a single task is higher on both (79 on R2R, 57.0 on REVERIE) [arxiv:2412.05552 tab. 4]. In the single-run ablations, batches that mix datasets cut R2R success from 73.69 to 62.36, so the paper keeps one dataset per batch [arxiv:2412.05552 tab. 6] [§4]. Without ScaleVLN pre-training, R2R success falls to 59.05 [arxiv:2412.05552 tab. 2]. All of these are author-reported, and tables 2, 3, 6 and 7 are single runs. The released code differs from the paper in several places, so the card's Paper vs code section should be read before any number is reproduced. Card: [SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts](/papers/2412.05552). ### Top-2 of 8 SwiGLU experts in every layer of a decoder-only model (Mixtral) Mixtral 8x7B replaces the feed-forward block of each layer of a Mistral 7B-style Transformer with a sparse layer of 8 SwiGLU experts [arxiv:2401.04088 §2] [tab. 1]. A linear router without bias picks 2 experts per token. The softmax is taken over those two logits only, so their weights sum to 1 [arxiv:2401.04088 §2.1]. The paper reports about 47B stored parameters, about 13B used per token, and a dense 32k context [arxiv:2401.04088 §3] [tab. 2]. The text describes no capacity limit, no token dropping and no load-balancing loss [arxiv:2401.04088 §2.1]. The paper's caption says Mixtral matches or beats Llama 2 70B on almost all of its benchmarks. Llama 2 70B is ahead on HellaSwag, WinoGrande and TriviaQA [arxiv:2401.04088 tab. 2]. On MMLU the scores are 70.6 for Mixtral against 69.9 for Llama 2 70B [arxiv:2401.04088 tab. 2]. The routing analysis finds no clear topic specialisation of experts [arxiv:2401.04088 §5] [fig. 7]. Card: [Mixtral of Experts](/papers/2401.04088). ### Top-2 of 128 experts with expert output masking in a translation model (NLLB-200) NLLB-200 replaces the feed-forward sub-layer of every fourth Transformer block with a sparse layer of 128 experts, and each token goes to its top two [arxiv:2207.04672 §8.2.2] [§6.2]. The paper gives 54.5B total parameters, with FLOPs similar to a 3.3B dense model [arxiv:2207.04672 §8.2.4]. The gate is a softmax over a linear map of the token [arxiv:2207.04672 §6.2] [eq. 7]. The training loss adds a load-balancing term with weight 0.01 [arxiv:2207.04672 §6.2] [eq. 9]. Each expert has a training capacity of 2 × ceil(T / E) for T tokens in the mini-batch and E experts. At generation, the capacity is set to the whole batch [arxiv:2207.04672 §8.2.4]. Expert output masking (EOM) is the paper's own regulariser. For a random fraction p of tokens, the first and/or second expert's output is zeroed during training. The final model uses p = 0.2 [arxiv:2207.04672 §6.2.2] [§8.2.4]. In the MoE-64 ablation on the dev set, EOM at p = 0.1 scores 44.7 chrF++ on the English-to-other pairs, against 44.3 for dropout at its best setting [arxiv:2207.04672 tab. 16]. Conditional MoE routing scores 44.8 in that table, slightly higher, but the authors state it costs 18 to 25% more training time, so they chose EOM [arxiv:2207.04672 §6.2.2] [tab. 16]. The ablations use a peak learning rate of 0.004, against 0.002 for the final model [arxiv:2207.04672 §6.2.1] [§8.2.4]. Card: [No Language Left Behind: Scaling Human-Centered Machine Translation](/papers/2207.04672). ### Dense gated blend of six experts in a motion VAE decoder (MVAE) This variant does not route tokens. The decoder of a motion VAE predicts the next pose of a character from the previous pose and a 32-dimensional latent [arxiv:2103.14274 §3.1] [§3.2]. The decoder has six experts, each with three layers of 256 ELU units. A gating network with 256 hidden units blends them [arxiv:2103.14274 §3.2]. The latent is fed into every layer of every expert, which the authors say limits posterior collapse [arxiv:2103.14274 §3.2] [§3.3]. The gate takes the latent and the previous pose, and the latent is used only in its first layer [arxiv:2103.14274 §3.2]. The expert count is ablated. The tables compare six and four experts, both at beta = 0.2 [arxiv:2103.14274 tab. 1]. Foot skate with six experts is lower than with four in four of the five columns: 0.067 against 0.085 on random walks, 0.28 against 0.39 on joystick, 0.30 against 0.44 on path following and 0.24 against 0.38 in the maze. The target column is the exception, at 0.27 against 0.24 [arxiv:2103.14274 tab. 1]. Four experts reach a target direction faster in the joystick test, at 1.38 seconds against 1.62 [arxiv:2103.14274 tab. 2]. These are single runs with no variance reported [arxiv:2103.14274 tab. 1]. The released code uses a gating width of 64, not the paper's 256, so the card's Paper vs code section should be read before the gate is implemented. Card: [Character Controllers Using Motion VAEs](/papers/2103.14274). ### Noise-level experts in a video diffusion model This variant is not a token router. The base is a 14B image-to-video MoE model, cited as Wan, with two experts: one for the high-noise steps and one for the low-noise steps [arxiv:2511.23429 §4.1]. The distillation timesteps are redefined around the noise boundary that separates the two experts, and each expert gets its own learning rate. The values are not stated [arxiv:2511.23429 §4.2.3]. In supervised finetuning only the MoE experts are trained, and the camera encoder is frozen [arxiv:2511.23429 §4.2.2]. Card: [Hunyuan-GameCraft-2](/papers/2511.23429). The paper's only ablation on the card covers cache and long-video settings [arxiv:2511.23429 fig. 16]. It does not isolate the effect of the expert split, so this variant has no measured benefit on this page. ## When to use which | Situation | Use | Why | |---|---|---| | Vision model, training from scratch, and more capacity is the goal | Top-2 MoE with Every-2 placement | The paper reports higher upstream and ImageNet fine-tuned scores than dense ViTs. Parameters rise from 656M to 7.2B for H/14, and so does training compute [arxiv:2106.05974 tab. 2]. | | Language model, more total capacity at a fixed per-token FLOP budget | GLaM-style top-2 of 64 experts in every other FFN layer | The paper reports better zero-, one- and few-shot averages than GPT-3 at about half its inference FLOPs per token [arxiv:2112.06905 tab. 1]. It does not state a capacity factor or the gating form, so an implementation must choose them [arxiv:2112.06905 §4] [§5.2]. | | Decoder-only language model with open weights, a fixed per-token compute budget and 8 experts per layer | Mixtral-style top-2 of 8 SwiGLU experts in every layer | The paper reports about 13B active parameters against 47B stored, and matches or beats Llama 2 70B on almost all of its benchmarks [arxiv:2401.04088 tab. 2]. The text describes no capacity limit or load-balancing loss, so an implementation must decide on both [arxiv:2401.04088 §2.1]. | | Many low-resource languages in one translation model | NLLB-style top-2 of 128 experts in every fourth block, with expert output masking | The paper's ablation favours EOM over plain dropout, by 0.4 chrF++ on dev, and the final model uses it [arxiv:2207.04672 tab. 16] [§8.2.4]. The headline results are on devtest and are author-reported [arxiv:2207.04672 §8.2.4]. | | Memory or communication is the bottleneck | Last-n placement, for example Last-5 for H/14 | Fewer expert layers and fewer parameters (2.7B against 7.2B). Scores drop only slightly in the paper's table [arxiv:2106.05974 tab. 2] [§3.1]. | | Routed-expert traffic dominates a large hybrid model | LatentMoE-style routing in a narrower latent space | The authors argue the saving scales with d/l and funds more experts at similar cost [arxiv:2604.12374 §2.1.1]. The paper's own serving claims are not measured against a common precision [fig. 1]. | | Image and text must share one encoder under contrastive training, and experts must not collapse by modality | LIMoE-style top-1 of 32 experts with per-modality entropy losses and batch priority routing | The paper reports average zero-shot gains of 10.1 points over dense one-tower models at matched compute [arxiv:2206.02770 §3.1]. The H/14 headline uses JFT-4B co-training and a soup [arxiv:2206.02770 §3.2]. Evaluating one modality alone needs capacity factor 16 [arxiv:2206.02770 app. D]. | | One trained model must serve several compute budgets | BPR, with C lowered at inference | No retraining is needed. The paper reports a smooth trade-off between accuracy and FLOPs [arxiv:2106.05974 §4.2]. Accuracy falls at very low C [app. C.4]. | | Each request's output must be reproducible on its own | Vanilla routing, or BPR with a fixed and reported batch | Under vanilla routing, overflow depends on token order within an image. Under BPR it depends on the batch [arxiv:2106.05974 §4.1]. The batch size used by the paper's tests is not stated. | | Experts risk going unused | Add a load-balancing term: weight 0.01 in V-MoE and GLaM | The V-MoE load term catches experts that are never selected, even when average router weights look balanced [arxiv:2106.05974 app. A.2]. GLaM uses the GShard form at 0.01 [arxiv:2112.06905 §5.2]. NLLB uses 0.01 [arxiv:2207.04672 §6.2]. Mixtral's text describes no such term [arxiv:2401.04088 §2.1]. Nemotron adds a bias update and a 1e-4 loss [arxiv:2604.12374 §2.4]. | | Low-precision training of an MoE model | Keep the router in FP32, and watch expert layers for underflow | Nemotron keeps its router in FP32 in the NVFP4 inference checkpoint [arxiv:2604.12374 tab. 7], and reports about 7% zero-valued weight gradients by the end of pretraining, traced to underflow in expert layers [arxiv:2604.12374 §2.2]. | | Starting from an existing MoE video base and adapting it | Noise-level experts, with the camera encoder frozen during finetuning | The authors finetune only the experts for their interaction task [arxiv:2511.23429 §4.2.2]. The expert split itself has no isolated measurement here. | | A multimodal model must learn tasks in sequence and cannot store old training data | LoRA experts with a constrained router and frozen-per-task experts, as in SAME | The paper reports the highest average accuracy in its CoIN and UCIT tables, including above Replay-LoRA on UCIT [arxiv:2602.01990 tab. 2] [tab. 3]. Its CoIN baselines are taken from the original papers, and its TriGap baselines are reproduced [tab. 1] [tab. 2]. The code's routing and scaling differ from the equations, so see the SAME card before implementing [SAME card](/papers/2602.01990). | | One navigation agent must handle several instruction styles, and mixing them in one model hurts | State-adaptive MoE on the visual query, with one dataset per training batch, as in SAME (navigation) | Single runs show the state router ahead of token and task routers on REVERIE [arxiv:2412.05552 tab. 2]. The unified model still trails the single-task ScaleVLN on R2R and REVERIE [arxiv:2412.05552 tab. 4]. | | An implementer needs a closed large MoE model's architecture | No variant on this page can be reproduced from its report | The Gemini 1.5 report names a sparse MoE and gives no expert count, routing top-k or layer count [arxiv:2403.05530 §3.1] [app. model card]. | | A motion or control model needs several sub-networks, and one latent drives all of them | Dense gated blend of six experts in the decoder, as in the MVAE | The paper reports lower foot skate with six experts than with four in four of five columns, in single runs [arxiv:2103.14274 tab. 1]. Its tables compare only MVAE variants with the motion capture data, so no single-network decoder is measured [arxiv:2103.14274 tab. 1] [tab. 2]. | | Audio, speech or any task other than image classification, language modelling, multimodal instruction tuning and video generation | No evidence on this page | The sources cover image classification in a ViT, language-model evaluation and pretraining, multimodal instruction tuning [arxiv:2602.01990 §5.1], machine translation [arxiv:2207.04672 §6.2], and video generation [arxiv:2106.05974 §1] [arxiv:2112.06905 §1] [arxiv:2511.23429 §1]. Nothing here is shown for other domains. | ## Implementation sketch The token-routing variants share one loop, with differences in the gate and in the overflow rule. V-MoE uses a softmax gate, fixed buffers, a visiting order and a capacity ratio C [arxiv:2106.05974 §2.3] [§4.1] [§2.4]. GLaM uses a softmax over experts, but its cards state no capacity limit [arxiv:2112.06905 §4]. Nemotron uses a sigmoid score with an expert bias [arxiv:2604.12374 §2.4] and states no capacity limit either. SAME uses a softmax gate over its LoRA experts and weights their outputs by the gate [arxiv:2602.01990 eq. 3]. It states no capacity limit, and the top-k cut appears only in its inference algorithm, without a value for k [arxiv:2602.01990 alg. 2]. Mixtral takes its softmax over the two kept logits only, so the weights sum to 1 [arxiv:2401.04088 §2.1]. NLLB takes a softmax over all experts and keeps the top two [arxiv:2207.04672 §6.2] [eq. 7], and its training capacity is 2 × ceil(T / E) [arxiv:2207.04672 §8.2.4]. LIMoE uses the same top-k gate and capacity rule, but it adds routing losses to the training objective and applies its priority rule during training [arxiv:2206.02770 §2.2] [app. B]. The noise-level variant has no router loop. Its expert is picked by the denoising step, and the card gives no code for that rule [arxiv:2511.23429 §4.2.3]. The navigation variant keeps the same gate form but routes once per sample per step, from a pooled view feature merged with the instruction token, and adds the selected experts' outputs to the visual query [arxiv:2412.05552 §3.1] [§3.2] [§3.4]. The motion VAE decoder is a third loop. It blends its six expert outputs with gate weights, and the paper describes no top-k cut, so every expert runs on each frame [arxiv:2103.14274 §3.2]. ```pseudocode # Sparse MoE layer over patch tokens [§2.1] [§2.3] [§2.4] # x: (N*P, D) tokens; E experts; k selected per token; C capacity ratio def moe_layer(x, W_router, experts, k, C, train): # Router logits, one per expert [§2.3] logits = x @ W_router # (N*P, E) # Training-only noise, std 1/E; test-time noise not stated [§2.3] if train: logits = logits + normal(0, 1 / E) # Softmax first, then keep the top-k experts [§2.3] gates = softmax(logits) # (N*P, E) # Buffer size per expert [§2.4] cap = round(k * N * P * C / E) # Renormalising the top-k gates: not stated [§2.3] fill = zeros(E) out = zeros_like(x) # Visiting order decides overflow: vanilla = row order [§4.1] # BPR = sorted by priority across the batch [§4.1] for t in visit_order(gates): for e in top_k_indices(gates[t], k): if fill[e] < cap: fill[e] += 1 # Gate-weighted sum of selected experts' outputs [§2.1] out[t] += gates[t, e] * experts[e](x[t]) # Overflow: this expert skips the token [§2.4] # The residual path carries the token forward [§2.4] return out ``` ## Contradictions - **BPR efficiency: text against tables.** The text says some models beat dense ViT-H while using under half the FLOPs and under 60% of the runtime [arxiv:2106.05974 §4.2]. The unmatched table it cites is built with vanilla routing, not BPR. In that table, H/14 Last-5 runs at 111.12% of dense FLOPs and 120.22% of dense time, and Every-2 at 135.59% and 164.89% [arxiv:2106.05974 app. C]. The FLOPs-matched BPR table lists the models that beat dense at about 100% FLOPs, with time at 118% to 156% [arxiv:2106.05974 app. C]. The paper also points to a low-capacity figure [arxiv:2106.05974 §4.2] [fig. 5]. This page did not check that figure. - **Upstream number for V-MoE-L/16 (Every-2).** The main table gives 57.65 JFT precision@1 [arxiv:2106.05974 tab. 2]. The appendix table, with vanilla routing, gives 57.64 with the same ImageNet 5-shot score of 77.10 [arxiv:2106.05974 app. C]. The paper does not explain the difference. - **SAME's router: paper against released code.** The paper's layer output sums each expert's gated update, computed per input [arxiv:2602.01990 eq. 3]. The released code routes once per batch, mixes the expert factors in a different way and uses a different expert rank [SAME card](/papers/2602.01990). The page reports the paper's equations, not the code's behaviour. - **SAME (navigation): text against its own table.** The text says lambda = 0.2 "performs worse than all other variants" [arxiv:2412.05552 §4.2]. In table 7, lambda = 0.2 gives the best R2R SPL and is near the best on ObjectNav SPL, but it has the lowest REVERIE success [arxiv:2412.05552 tab. 7]. The claim holds for some metrics only. The released code also differs from the paper's softmax axis and balance-loss scale [SAME navigation card](/papers/2412.05552), so the two sources are not interchangeable. - **Compute baselines differ.** GLaM reports lower inference FLOPs and training energy than GPT-3 [arxiv:2112.06905 tab. 1]. V-MoE's table shows more training compute than the dense ViT-H/14 [arxiv:2106.05974 tab. 2]. The two papers compare against different dense models and in different domains, so their compute claims are not directly comparable and are not reconciled here. ## Papers - [Scaling Vision with Sparse Mixture of Experts](/papers/2106.05974): introduces V-MoE, top-k routing with capacity limits in a ViT, and Batch Prioritized Routing for inference. - [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](/papers/2112.06905): a decoder-only language model with 64 experts per MoE layer and top-2 routing, compared with GPT-3 on zero-, one- and few-shot tasks. - [Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](/papers/2604.12374): a hybrid Mamba-Transformer with 512 experts and top-22 routing in a latent space, pretrained with NVFP4 GEMMs. - [Mixture of Horizons in Action Chunking](/papers/2511.19433): listed under this concept, but its gate mixes horizon predictions rather than experts. See the Overview. - [SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning](/papers/2602.01990): mixture of LoRA experts in the feed-forward layers of a multimodal LLM, with constrained router and expert updates for sequential tasks without replay. - [SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts](/papers/2412.05552): a mixture-of-experts layer on the visual query of the DUET navigation agent, routed by the agent's state, trained once on seven navigation datasets. - [Character Controllers Using Motion VAEs](/papers/2103.14274): a motion VAE whose decoder blends six experts with a gate, with an RL policy that steers its latent. - [Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model](/papers/2511.23429): uses a 14B MoE video base with high- and low-noise experts, inside a larger distilled game-world system. - [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](/papers/2403.05530): a closed report that calls Gemini 1.5 Pro a sparse MoE Transformer, with no expert count or routing details. - [Multimodal Contrastive Learning with LIMoE](/papers/2206.02770): one shared image-text transformer with top-1 routing over 32 experts, per-modality entropy losses and batch priority routing during training. - [Mixtral of Experts](/papers/2401.04088): a decoder-only language model whose layers each hold 8 SwiGLU experts, with a top-2 router and open weights. - [No Language Left Behind: Scaling Human-Centered Machine Translation](/papers/2207.04672): a 54.5B translation model with 128 experts in every fourth block, top-2 routing, and expert output masking.