Oxford's GauntletBench puts AI agents through 100 real tasks. They failed 81% of them.

As AI agents increasingly find their way into real-world settings, the need for rigorous testing has never felt more pressing. Researchers at the University of Oxford have launched GauntletBench, a new benchmark designed to bypass the fatigue that has settled over the field of agent evaluation. It homes in on three underexplored capabilities, temporal perception, graphical understanding, and 3D reasoning, across five professional applications that existing suites have largely ignored.

The benchmark, published on arXiv and hosted on HuggingFace, puts AI agents through 100 vision-intensive tasks organized into five categories: Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Designer. Each one simulates a controlled web environment where an agent can click, type, and navigate. The modular pipeline supports both open- and closed-source agent frameworks.

Reality check for agentic systems

The top-performing agent, built on a frontier large language model, managed a success rate of just 19.1% across all tasks. Non-expert human annotators completed the same tasks with over 80% accuracy, a figure the paper describes as “challenging yet feasible” for people. That gap, the researchers argue, stems from the narrow focus of most existing benchmarks. Many current tests focus on everyday consumer apps, shopping, travel booking, or text-heavy tasks that play to the strengths of language models. GauntletBench deliberately avoids these well-trodden paths. Its tasks require agents to work through dynamic video timelines, interpret 2D diagrams, manipulate 3D objects, analyze flight paths, and reason about circuit schematics, all through a visual interface.

Three critical capability gaps

GauntletBench highlights three specific areas where agents repeatedly come up short:

• Temporal perception: Agents stumble when they need to track events over time, adjusting a video edit timeline, for instance, or monitoring a sequence of workflow steps. The failures suggest a lack of robust temporal reasoning beyond simple sequence prediction.
• Graphical understanding: Tasks that involve reading and manipulating 2D visual representations, flowcharts, circuit diagrams, layout grids, proved dicey. Agents often misread spatial relationships or failed to map visual elements to their functional roles.
• 3D reasoning: In the 3D Modeller application, agents had trouble rotating objects, understanding depth, and executing precise spatial operations. It mirrors broader findings that multimodal models still lack reliable 3D comprehension.

These results dovetail with ongoing conversations in the field about the limits of current multimodal AI systems. While large language models shine at text-based reasoning, their ability to understand and act on visual and temporal signals remains brittle.

Structure and extensibility

GauntletBench is built as a modular pipeline with four components: an environment layer that interfaces with web-based applications, a controlled application layer that simulates the five professional tools, a task suite of 20 tasks per application generated from templates, and an automated evaluation engine that calculates success using both exact and partial matching criteria. That design allows researchers to add new applications or task variants without rebuilding the pipeline.

“Even the state-of-the-art agent achieves only a 19.1% success rate on our GauntletBench, highlighting the limitations in these overlooked capabilities and generalisation,” the authors write in the paper.

The choice of professional applications, video editing, workflow automation, 3D modelling, flight analysis, and circuit design, is deliberate. These domains demand precise, multi-step interactions with complex visual interfaces, and they have been underrepresented in prior agent benchmarks, which the research community had flagged as saturated.

What it means for agent design

The wide gap between agent and human performance, more than 60 percentage points, suggests that current approaches to agent architecture, training, or prompting aren't up to the task of handling complex visual-temporal jobs. The paper calls for a renewed focus on developing agents that can generalize across unfamiliar applications and modalities. The benchmark doesn't prescribe specific architectural changes, but it does provide a detailed failure analysis. Agents fared better on tasks with clear visual feedback, a color change after a successful action, than on tasks demanding delayed reasoning, like predicting the outcome of a multi-step editing sequence.

Oxford's contribution arrives at a time when the AI research community is taking a harder look at benchmark design. Several recent papers have flagged benchmark contamination, task oversimplification, and task homogeneity as obstacles to meaningful agent evaluation. GauntletBench takes direct aim at those issues by introducing novel task types and high-difficulty scenarios.

The benchmark has already picked up attention on HuggingFace, where the project paper earned 15 upvotes and positive feedback from the community. The researchers plan to expand GauntletBench to include more applications and longer-horizon tasks in future releases.

For agent developers, the message is clear: the easy parts of the problem are solved. The remaining challenges, temporal, graphical, and spatial reasoning, demand fundamentally new approaches if AI agents are going to operate reliably in the wild.