Ai2's olmo-eval gives LLM developers a microscope for every checkpoint

The Allen Institute for AI (Ai2) has released olmo-eval, an open evaluation workbench built for the daily grind of developing large language models. The framework extends the Open Language Model Evaluation Standard (OLMES) introduced in 2024, which aimed to make benchmark scores comparable across model releases.

Designed for ongoing model development

Most existing evaluation tools, according to Ai2, are built either to run established benchmarks on finished models or to test agents in sandboxed environments. Neither fits the reality of active development, where researchers make frequent adjustments to data, architecture, or hyperparameters and need quick feedback on each new checkpoint. Olmo-eval was built specifically for this loop, letting developers add benchmarks, configure runtime policies, and compare results question by question rather than relying solely on aggregate scores.

“A model’s final score is only part of the evaluation process,” said Ai2 researchers in the announcement. Olmo-eval builds on OLMES by reducing the work needed to implement new evaluations and offering more flexibility in how they run.

Modular architecture: tasks, harnesses, and suites

The framework has four main components. A task defines what is being evaluated: the benchmark dataset, how evaluation requests are built, and how answers are scored. A suite groups related tasks for joint execution. A harness controls the runtime policy, including the model, tools, scaffolding, and sandbox environment. This separation means the same task can run under different conditions without rewriting code.

A basic question-answering benchmark, for example, can be defined in a few lines of Python using simple dataloaders and formatters. A variant can tweak parameters such as few-shot prompting. A suite bundles multiple benchmarks into one command. The same benchmark can run with or without tool access, enabling direct comparison of how a model performs with and without search or code execution.

Sandboxed execution for agent evaluations

Olmo-eval includes a sandbox and capability-routing layer for evaluations where models must use tools such as writing and running code, browsing the web, or interacting with APIs. Unlike frameworks that force every evaluation into a fully containerized environment, olmo-eval picks the lightweight default: a benchmark that only needs a model to answer questions runs directly. When the benchmark requires a locked-down environment, for example to execute generated code safely, an isolated container spins up.

This design, Ai2 says, makes the tool faster and cheaper for standard evaluations while preserving security and reproducibility for more complex agentic workflows.

Comparison with Harbor

Ai2 explicitly contrasts olmo-eval with Harbor, an open framework for evaluating AI agents inside sealed containers. Where Harbor focuses on publishing agent benchmarks with extra verification steps for public sharing, olmo-eval is geared toward everyday model development. Harbor’s process for adding benchmarks is built for public sharing; olmo-eval prioritizes speed and flexibility during development.

Olmo-eval’s modularity also lets developers swap the model, tools, helper models such as an LLM-as-a-judge, and environment settings independently. A tool definition written with the @tool decorator can be reused across multiple harnesses without duplication.

Granular comparison: scores with standard error

Olmo-eval reports overall scores with standard error and a minimum detectable effect, but its most distinctive feature is a per-question pairwise comparison view. This lines up the same questions across two model checkpoints and compares answers one by one, holding all other variables fixed. The goal is to help developers see whether a small change in an overall average reflects a real improvement or just statistical noise.

“If your recurring question is 'How does this checkpoint differ from the last one, and where exactly did it improve or regress?', that’s the workflow olmo-eval is built for,” Ai2 stated.

Open release and integration with Ai2’s model line

Olmo-eval builds directly on OLMES, the standard Ai2 introduced in 2024 to make benchmark scores comparable across model releases. The OLMES standard was previously used to evaluate Ai2’s open models, from Olmo to Tulu. The new workbench extends that approach into the active development phase.

The framework is available as open-source software, and Ai2 invites the community to contribute tasks, suites, and harnesses.