Sandbox benchmarks are hiding how agents really fail, HKU just built the fix

Benchmarking AI agents has a quiet deception built into it. Most evaluations run in sandboxed environments, carefully trimmed gardens where the agent never has to install a package, interpret a blurred screenshot, or recover from an unexpected server reboot. The result is a rosy picture: agents score high, but fail the moment they touch an unscripted operating system.

Researchers at the University of Hong Kong say this mismatch is not just a methodological annoyance. It actively misdirects development. Their new benchmark, UniClawBench, is designed to tear down the sandbox walls and test agents where they break: inside real, live Docker containers, executing 400 bilingual tasks that span five foundational capabilities. Your AI search pipeline is broken. This open-source…

Five capabilities, not fifty scenarios

Existing benchmarks typically use scenario-based taxonomies: web browsing, calendar scheduling, file editing. The problem, according to the team, is that each scenario depends on multiple overlapping model skills. When an agent fails, it is unclear whether the root cause is weak long-context reasoning, poor multimodal understanding, or insufficient tool usage ability.

UniClawBench sidesteps this by building its 400 tasks around exactly five capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Each task is designed to test one capability predominantly, with minimal skill blending. This lets failure analysis pinpoint the exact weak link in the model. Jet-Long's bifocal attention just killed the… ViQ just gave multimodal AI the one thing it needed:…

Evaluation on live machines

Where UniClawBench diverges most sharply from prior work is execution. Instead of scoring pre-recorded answers against static datasets, the benchmark deploys agents inside live Docker containers. The agent must interact with the actual filesystem, network, and application APIs. Nothing is simulated. Completion is measured via fine-grained, step-by-step checkpoints that a hidden supervisor agent monitors in real time.

"This is the difference between a driving test on a closed track and one on city streets," the paper notes. "The agent must handle unexpected states, missing dependencies, and system responses that no pre-recorded answer can capture."

A closed loop with three roles

One of the hardest problems in evaluating proactive agents, those that initiate actions rather than just react to queries, is simulating human feedback without leaking the correct answer. HKU's solution is a three-actor evaluation loop: an executor agent, the model under test; a user agent that plays the role of a person requesting tasks; and a hidden supervisor agent that checks intermediate checkpoints and provides progressive feedback.

The user agent is deliberately fallible. It may ask clarifying questions, change its mind mid-task, or ignore incomplete results. This forces the executor to manage multi-turn interactions dynamically. Crucially, the supervisor agent never reveals future checkpoints. It only confirms or denies progress on the current one, preventing answer leakage. Your AI assistant forgets you every morning. This…

Models and frameworks disentangled

A central contribution of UniClawBench is its factorial design: each state-of-the-art model is evaluated under multiple agent frameworks. The benchmark can decompose performance into a model's raw capability score and a framework's orchestration efficiency. The early results, while not detailed in the preprint, show that a strong model in a weak framework can be outperformed by a weaker model in a smart framework. That finding has direct implications for both model training and system design.

The five-capability structure also reveals surprising intra-model asymmetries. A model that excels at Long-Context Reasoning may collapse on Cross-Platform Coordination, even though both tasks fall under the same "web agent" scenario in conventional benchmarks. These asymmetries are exactly the kind of signal development teams need to target their optimization efforts. Why GPT-5.5 dominates a benchmark that tests how agents…

Implications for the agent ecosystem

For startups building agentic products, UniClawBench offers a diagnostic tool rather than a leaderboard. It answers not just "which agent is best" but "which capability is the bottleneck for each model-framework pair." For the research community, the benchmark provides common ground to compare not just final scores but failure patterns across the five dimensions.

The benchmark is released open-source alongside its code, with tasks in both Chinese and English. The HKU team emphasizes that UniClawBench is designed to evolve. New tasks and capabilities can be added without altering the evaluation infrastructure, making it a living benchmark for a fast-moving field.