GLOSSARY
Mixture of Experts (MoE)
An architecture that routes each request to a small subset of specialized sub-networks — frontier-level quality at a fraction of the compute per token.
A dense model activates every parameter for every token. A mixture-of-experts model contains many parallel “expert” sub-networks and a router that activates only a few per token — for example 8 experts out of 256. The model's total knowledge is huge, but the compute per token stays small.
This is the architecture behind many of the strongest open and commercial models of the current wave. It explains a common confusion: a model advertised as “671B parameters” may cost less to serve than a dense 70B, because only a fraction of those parameters run for any given request.