Mixture of experts (MoE)
Architecture where only a subset of the model's parameters ("experts") activates for each input, cutting compute; nearly every 2026 frontier open model uses it.
In a mixture-of-experts model, a router sends each Token to a few specialized sub-networks (“experts”), so only a fraction of parameters is active per token. That is why 2026 models are quoted as two numbers: Kimi K3.1 Pro is 2.8T total / 104B active, Mistral Large 4 is 1T / 49B, Beam is 501B / 23B. Total parameters drive memory needs; active parameters drive per-token Inference cost.
Timeline
Every dated event in the wiki that involves Mixture of experts (MoE), newest first.
- Shanghai AI Laboratory releases Sage S2.1 Flash, an MIT-licensed 321B Open-weight vs. closed models model for scientific research
The mixture-of-experts model from the InternLM team targets multi-round scientific research, with an FP8 variant and a separate perception projector for scientific inputs. Its card claims first place on 4 of 15 benchmarks against six models, including MLE-bench Lite (90.3), while it trails Claude Opus 5 on SWE-bench Pro (60.6 vs 76.5); all figures are self-reported.
Sources: Hugging Face, “internlm/Sage-S2.1-Flash”
- Thinking Machines Lab releases Inkling 2 Small
594B-total / 17B-active MoE with image input; closed weights on Tinker at launch, Apache 2.0 weights promised after external safety testing. Vendor-cited Artificial Analysis Intelligence Index score: 31.
- Baidu releases ERNIE-5.5-Flash-Base
Open-weight 280B base MoE checkpoint with no post-training.
Sources: The Decoder, “Baidu releases ERNIE 5.5 Flash Base with 280 billion parameters as an open-source foundation model” (citing Baidu's release page)
- Moonshot AI releases Kimi K3.1 Pro
2.8T MoE with 104B active, 1M context, $2/$10 per million tokens; open weights pledged by Nov 4.
Sources: kimi.ai; MarkTechPost; Reuters
- Reflection AI releases Beam open weights
501B sparse MoE under Apache 2.0; 80.1% on Terminal-Bench 2.1 and 81.6% on SWE-bench Verified per the model card, with a claim of 3-4x less inference compute than GLM-5.2 (self-reported).
Sources: BenchmarkList, “Beam 1.0 Benchmark Scores & Evals”; VentureBeat
- Meta releases Muse Spark Open
816B MoE (42B active) open-weight model under the Muse Model License with 1M context.
Sources: Meta AI Research
- Microsoft AI publishes MAI-Base-2
2.8T MoE (80B active) trained on 16k GB200s; weights for trusted partners via Foundry.
Sources: Microsoft AI
- StepFun releases Step 5 Preview
600B sparse MoE with 1M context on OpenRouter.
Sources: OpenRouter, “StepFun: Step 5 Preview”; OpenRouter listing
- Mistral previews Mistral Large 4 "Le Chonk"
1T-parameter natively multimodal MoE with 52B active parameters (Mistral; some coverage gives 49B excluding embeddings); preview API, weights promised by month end.
Sources: Mistral, “Introducing Mistral Large 4”
- Z.ai open-sources GLM-5.2
MoE (40B active; 744B per TechCrunch, 753B per VentureBeat) under MIT with 1M context.
Sources: Let's Data Science, “Z.ai Releases GLM-5.2 With 1M-Token Context”; Trending Topics, “GLM-5.2: Zhipu AI aus China schlägt mit offenem Modell sogar Googles Top-Modelle”
Connections
Mixture of experts (MoE) and everything one link away. Drag to move, scroll to zoom, click a node to open it.Open in the full graph →
What links here
AI lab Baidu (ERNIE) · DeepSeek · IQuest · Meta · Microsoft AI (MAI) · Mistral AI · Moonshot AI (Kimi) · Recursive · Reflection AI · StepFun · Thinking Machines Lab · Z.ai (Zhipu AI)
Model family DeepSeek models · GLM · Kimi · Muse (Meta models)
Concept Inference
Sources
- Wikipedia, “Mixture of experts”