Token-Level Analysis Reveals Hybrid LLMs Excel at Meaning-Bearing Words, Lag on Repetitions

A hybrid language model can match or beat a standard transformer on aggregate benchmarks, but what does that headline number hide? A team at the Allen Institute for AI set out to answer that question by comparing two closely matched 7B-parameter models, Olmo 3, a pure transformer, and Olmo Hybrid, which replaces most attention layers with recurrent layers, at the token level.

Attention vs. Recurrence: Complementary Strengths

In a transformer, every layer uses attention, letting each token draw directly on all earlier tokens. That makes attention excellent at recalling specific tokens from far back, but its computational cost scales quadratically with input length. A hybrid model keeps a few attention layers but replaces the rest with recurrent layers that carry a fixed-size compressed memory, processing each token at constant cost regardless of input length. That compressed memory is lossy, so recurrent layers cannot reach back for exact copies, but they excel at tracking information that evolves sequentially.

Methodology: The Loss Gap

To isolate where each architecture wins, the researchers fed both models the same passages of text, articles, Wikipedia entries, books, scientific papers, Python, HTML, and LaTeX, and recorded the probability each assigned to the actual next token. They computed the loss gap (the difference in loss) token by token: a positive gap means the hybrid predicted better, a negative gap means the transformer did.

"We then sort each token into a category and average the loss gap within these categories, then re-check each pattern with a regression that estimates the category's own effect while holding other factors constant," the team explained.

Key Findings: Content Words vs. Function Words

In prose, the clearest divide is between content words, meaning-bearing nouns, verbs, and adjectives, and function words like "the," "of," and "is." The hybrid predicts content words significantly better than the transformer, with a loss gap around 0.03, while the gap narrows to near zero on function words. The advantage is especially pronounced on adverbs and adjectives.

"The hybrid's edge is biggest on the words that say what a sentence is about and smallest on the grammatical words any model can nearly guess from syntax," the researchers noted.

Where the Hybrid's Advantage Disappears

The hybrid's lead all but vanishes on tokens that simply repeat something already in the input. The team identified these cases by looking for repeated n-grams, runs of text where the token completing a sequence has appeared verbatim earlier. The longer the repeated run, the smaller the hybrid's lead, until it approaches zero.

Another clear case is closing braces. The team found that closing, but not opening, braces show no hybrid advantage across brackets in language, code, and markup, a pattern consistent with attention being sufficient for bracket matching.

Filtered Token Losses as an Evaluation Tool

Building on these findings, the team explored using filtered losses on specific token types as an evaluation method. They compared three 1B-parameter models: a transformer, a hybrid, and a pure recurrent model with no attention at all.

On meaning-bearing tokens that aren't repeats, both the hybrid and the pure recurrent model overtake the transformer, with the hybrid performing best. On repeated tokens, the pure recurrent model, with no attention to reach back for the copy, falls behind both the hybrid and the transformer.

"Thus, these filtered token losses reveal different fine-grained differences between architectures, including copying abilities and differences on content words, early in training in a way that would not otherwise be visible," the researchers wrote.

Implications for Architecture Design

The work carries two main lessons. First, a single overall loss, the model's average error across all tokens, is too blunt to compare transformer and hybrid architectures. Scoring the loss on just the tokens that test a specific ability surfaces key differences.

Second, the hybrid's particular advantage on open-class tokens may relate to the state-tracking capabilities of recurrent layers. "We believe the best hybrid architectures will come from understanding, token by token, what each component of a model does well," the team concluded.

The researchers have released the full report, Olmo 3, Olmo Hybrid, and their associated open artifacts for the community to explore.