Mixture of Experts

Also: MoE

An architecture where different subsets of model parameters ('experts') activate for different inputs — enabling very large models without proportionally large compute costs.