AI Wiki · Mediocre Security
Concept

Mixture of experts (MoE)

Architecture where only a subset of the model's parameters ("experts") activates for each input, cutting compute; nearly every 2026 frontier open model uses it.

In a mixture-of-experts model, a router sends each Token to a few specialized sub-networks (“experts”), so only a fraction of parameters is active per token. That is why 2026 models are quoted as two numbers: Kimi K3.1 Pro is 2.8T total / 104B active, Mistral Large 4 is 1T / 49B, Beam is 501B / 23B. Total parameters drive memory needs; active parameters drive per-token Inference cost.

Timeline

Every dated event in the wiki that involves Mixture of experts (MoE), newest first.

  • Releasemodel card undated; reported Nov 18
    Shanghai AI Laboratory releases Sage S2.1 Flash, an MIT-licensed 321B Open-weight vs. closed models model for scientific research

    The mixture-of-experts model from the InternLM team targets multi-round scientific research, with an FP8 variant and a separate perception projector for scientific inputs. Its card claims first place on 4 of 15 benchmarks against six models, including MLE-bench Lite (90.3), while it trails Claude Opus 5 on SWE-bench Pro (60.6 vs 76.5); all figures are self-reported.

  • Release
    Mistral previews Mistral Large 4 "Le Chonk"

    1T-parameter natively multimodal MoE with 52B active parameters (Mistral; some coverage gives 49B excluding embeddings); preview API, weights promised by month end.

Connections

Mixture of experts (MoE) and everything one link away. Drag to move, scroll to zoom, click a node to open it.Open in the full graph →

What links here

AI lab Baidu (ERNIE) · DeepSeek · IQuest · Meta · Microsoft AI (MAI) · Mistral AI · Moonshot AI (Kimi) · Recursive · Reflection AI · StepFun · Thinking Machines Lab · Z.ai (Zhipu AI)

Model family DeepSeek models · GLM · Kimi · Muse (Meta models)

Concept Inference

Sources

  1. Wikipedia, “Mixture of experts”