Ai2's EMO makes modular AI emerge from data, not human rules

Ai2 has released EMO, a new mixture-of-experts (MoE) language model pretrained end-to-end so that modular structure emerges directly from the data, no human-defined priors required. The model lets users activate only a small subset of its experts, just 12.5% of the total, for a given task while retaining near full-model performance. Yet it also works as a strong general-purpose model when all experts are combined.

Large language models are typically trained and deployed as monolithic systems: a single model is initialized, pretrained, fine-tuned, and served as one unified entity. But applications often need only a subset of capabilities, code generation, mathematical reasoning, or domain-specific knowledge. As frontier models routinely reach trillions of parameters, using and adapting the full model becomes impractical for most users and incurs unnecessary computational cost and memory.

MoE models seem like a natural way to relax this constraint. Instead of using one large feedforward network at each layer, MoEs contain many smaller ones, called experts, and activate only a small subset for each input token. In principle, a task that only needs one capability could load only the relevant experts.

In practice, however, existing MoEs still need the full model to work well. Even within a single input, different tokens often activate different experts, so a task can end up using all the experts during its generation. As Ai2 researchers show in the EMO paper, this happens partly because experts in standard MoEs often specialize in low-level lexical patterns, think prepositions or punctuation, rather than higher-level domains or capabilities. As a result, small subsets of experts are not reliably usable on their own.

Ai2 instead wanted MoE models whose experts organize into coherent groups that can be selectively used and composed.

How EMO achieves emergent modularity

One way to encourage this during pretraining is to route tokens to experts based on predefined semantic domains, such as math, biology, or code. Prior work like BTX and Ai2's FlexOlmo project has tried this. However, predefined domains come with important limitations. They require domain labels across the pretraining corpus, which can be ambiguous and expensive to obtain, and may inject too much human bias into how the model is allowed to organize itself.

EMO takes a different approach. The key insight is that tokens from the same document usually come from the same domain. Ai2 therefore uses document boundaries as a weak supervisory signal: during training, all tokens in a document are restricted to choose their active experts from a shared expert pool.

In an MoE with 10 total experts and 2 active experts per token, all tokens in a document are restricted to route within the same pool of 4 experts. This pool is chosen by the router itself: it averages the router's expert preferences across all tokens in the document, then selects the most-used experts as the document's shared pool. Different documents can use different pools, allowing recurring expert groups to emerge directly from the training data.

The researchers note several implementation considerations:

• Load balancing: In standard MoE training, the load-balancing objective prevents the model from collapsing onto only a small number of experts. EMO applies load balancing globally across many documents, making the two objectives complementary: EMO encourages tokens within the same document to use a coherent expert pool, while global load balancing encourages different documents to collectively cover all experts.
• Document pool size: Rather than fixing one pool size, Ai2 randomly samples it during training. This prevents EMO from overfitting to a single subset size and lets it support different expert subset sizes at inference time.

Benchmark results and performance

On general-purpose benchmarks, EMO matches the performance of a standard MoE model, showing that the modularity objective does not come at the cost of full-model performance. The more important question, however, is whether the model can still work when only a subset of experts is kept. In this setting, Ai2 constructs task-specific expert subsets by ranking experts according to their routing usage on a small amount of task validation data, keeping the most-used experts and discarding the rest.

The results show that EMO remains robust under selective expert use. When keeping only 25% of the experts (32 expert subset), EMO loses only about 1% absolute performance across all benchmarks; even when keeping only 12.5% of the experts (16 expert subset), the overall drop is only about 3%. This holds both before and after fine-tuning. In contrast, the matching standard MoE degrades sharply as the expert subset gets smaller, often falling close to or below random performance in the smallest expert subset settings.

Furthermore, Ai2 shows that selecting the right experts for a task is surprisingly cheap: a single example with few-shot demonstrations is enough to identify a module that performs on par with one selected using a full validation set. EMO also works well with existing expert-pruning approaches like Easy-EP, and the two complement each other.

What expert subsets specialize to

To see what EMO actually learned, Ai2 researchers clustered router activations of the first 100 tokens across 12,000 pretraining documents. The difference from a standard MoE is stark.

EMO's token clusters correspond to things like Health, Medical & Wellness, News Reporting, US Politics & Elections, and Film & Music. A standard MoE produces clusters like Prepositions, Proper Names, Copula Verbs, or Definite Articles. In EMO, tokens from a given document mostly land in the same cluster; in a standard MoE, they end up scattered across many.

The contrast is easiest to see on a single example. Take a health article: in EMO, almost every token would route into the Health, Medical & Wellness cluster. In a standard MoE, the top cluster is Possessives & Definite Articles; the model would group the article with every other text that happens to use the word 'the' or 'your,' regardless of what that text is about.

Because EMO forms modules that map to semantic domains rather than surface features, users can pick a small expert subset and still have a functioning model, the group corresponds to a real capability.

Release and future work

Ai2 is releasing the full EMO-trained model, a matched standard-MoE baseline trained on the same data, and the training code. The researchers hope these artifacts are useful for other groups studying emergent modularity in MoEs.

There is more work to do. EMO is an early step toward making large sparse models more modular, but many questions remain: how to better select and compose expert subsets, how to update modules without disrupting the full model, and how to use modular structure for better interpretability and control. Releasing these models should help the community study these questions and build toward modular language models that are easier to deploy, adapt, inspect, and compose.