An architecture that routes inputs to specialized sub-networks, enabling massive scale efficiently.
Mixture-of-Experts (MoE) is a neural network architecture where the model consists of many 'expert' sub-networks and a learned 'router' that decides which experts to activate for each input token. Only a small fraction of experts are active at any time, making MoE models far more computationally efficient than dense models of the same parameter count. LLaMA 4-400B uses MoE — activating ~40B parameters per token despite having 400B total. GPT-4 is widely believed to be a MoE model internally. MoE is now considered essential for building frontier-scale models economically.
Get stories like these every Monday. No noise, just signal.
12,400+ subscribers. Unsubscribe anytime.