The benchmark that made language models speak: how 2018's glue bet changed ai forever

Seven years ago, a group of researchers at New York University and DeepMind published a paper that most people ignored at first. Titled 'GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding,' it proposed a simple but radical idea: instead of each lab testing its models on a different set of tasks, the field needed a common yardstick.

That yardstick became the General Language Understanding Evaluation benchmark, GLUE. It would go on to become the most cited NLP benchmark in history, with over 10,000 citations, and its successor SuperGLUE would define the race toward human-level language AI.

The problem GLUE solved

Before 2018, NLP research was fractured. Teams tested their models on whatever datasets they had access to, sometimes just one or two tasks. Results were hard to compare. A model that scored well on sentiment analysis might fail at question answering, but nobody had a systematic way to measure that trade-off.

GLUE assembled nine diverse tasks into a single score. They ranged from grammatical acceptability judgments (CoLA) to natural language inference (MNLI, QNLI, RTE), paraphrase detection (MRPC, QQP), sentiment analysis (SST-2), and textual similarity (STS-B). The tasks intentionally spanned different linguistic phenomena, syntax, semantics, pragmatics, and different dataset sizes, from a few thousand examples to hundreds of thousands.

'The idea was to create a stress test,' said one of the benchmark's creators in a 2020 interview. 'If your model can do well on all nine tasks, you have some evidence that it's actually learning language, not just memorizing patterns in a single dataset.'

The benchmark included a held-out test set on a private server, preventing accidental overfitting. And it enforced a strict rule: participants could only submit once per task per week, discouraging brute-force tuning.

The race to the top

GLUE's leaderboard quickly became the most competitive arena in AI. In 2018, the state of the art hovered around 70 points out of 100. Then came ELMo, BERT, RoBERTa, XLNet, ALBERT, and T5, each model leapfrogging the previous one. By mid-2019, BERT Large scored 82.7. By 2020, T5 pushed past 90. And by 2022, models were regularly scoring above 92, approaching what the benchmark's creators estimated as human performance: around 87.1 to 92.8, depending on the task.

The benchmark's competitive dynamic did exactly what its creators hoped: it concentrated research effort on a concrete, quantifiable goal. 'GLUE turned NLP into a sport,' remarked a Google researcher in 2019. 'Every lab wanted to be at the top of that leaderboard.'

But the sport had side effects. As models climbed the leaderboard, researchers began questioning whether GLUE actually measured language understanding, or just the ability to exploit statistical shortcuts in the benchmark's datasets.

SuperGLUE and the reckoning

By late 2019, GLUE had reached a saturation point. Many models scored above 90, and the hardest tasks had been effectively solved. The benchmark's creators released SuperGLUE in November 2019, a harder version with eight new tasks designed to resist the same pattern-matching tricks. These included reading comprehension with commonsense reasoning (ReCoRD), winograd-style pronoun resolution (WSC), and multi-sentence reading comprehension (MultiRC).

SuperGLUE reset the race. Initial scores on the new benchmark were in the 60s. By 2021, models reached the high 80s. And when PaLM 2 scored 91.3 in 2023, it surpassed the estimated human baseline for the first time, though the benchmark's designers cautioned that human performance estimates were themselves uncertain.

'SuperGLUE is harder, but it inherits the same fundamental tension as GLUE: the desire for a single number that encapsulates progress versus the reality that language understanding is multidimensional,' wrote one NLP researcher in a 2022 survey.

The legacy: from GLUE to the present

GLUE's influence extends far beyond its leaderboard. It established the template that virtually every subsequent AI benchmark follows: a curated set of tasks, a weighted aggregate score, a secret test set, and a public leaderboard. The Hugging Face datasets library, which hosts hundreds of NLP datasets, owes its structure partly to the standardized format GLUE pioneered.

Today's most prominent AI benchmarks, MMLU for knowledge, HumanEval for code, Chatbot Arena for conversational ability, all borrow elements from the GLUE paradigm. They share the same tension between standardization and simplification: they make progress measurable but risk narrowing what gets optimized.

The forgotten critique

GLUE's most important contribution may be the critique it created. The benchmark successfully highlighted how easily models could achieve high scores without genuine understanding, a problem that had been present since the earliest days of AI evaluation. In 2019, researchers at Facebook AI showed that BERT could be misled by surface-level patterns, such as changing a single negation in MNLI examples. The benchmark became a laboratory for studying these vulnerabilities, which later informed work on adversarial robustness and dataset contamination.

When GPT-3 was released in 2020, the authors explicitly benchmarked it against SuperGLUE, reporting a score of 71.8 with few-shot prompting. That number, impressive but not dominant, helped frame the conversation about GPT-3's capabilities as impressive but not revolutionary for language understanding tasks. It gave the community a common reference point.

What GLUE didn't capture

For all its impact, GLUE left out entire dimensions of language understanding. It tested no long-form generation, no dialogue coherence, no reasoning about temporal or causal relationships, and no linguistic grounding in the physical world. Tasks were presented in English only. The benchmarks implicitly assumed that language understanding was a property of text alone, not of the world text refers to.

These gaps explain why later benchmarks evolved in different directions: MMLU tests world knowledge, TruthfulQA tests honesty, and BigBench tests structured reasoning. But none of them would exist in their current form without GLUE proving that the benchmark-first approach could work.

The quiet retirement

GLUE's website still hosts the leaderboard, but the benchmark is largely retired from active use. The last significant score was posted in 2022. Most labs now test on MMLU, HumanEval, or custom evaluations. When Anthropic released Claude 3 in 2024, they reported MMLU scores, not GLUE. When Google published Gemini, they led with MMLU.

Yet every new benchmark faces the same fundamental challenge GLUE identified a decade ago: how to design an evaluation that is rigorous enough to be useful, broad enough to be meaningful, and difficult enough to resist exploitation. The answer is never permanent. But the conversation GLUE started continues to shape the field.

As one of its creators put it in a 2023 retrospective: 'We built GLUE because we were frustrated that nobody could compare models. We never imagined it would become the template for the next decade of AI evaluation. We just wanted to know which model was better, and language is so complex that even that simple goal turned out to be a multi-year research project.'