The real bottleneck in desktop AI agents isn't the model. It's the skill library.

Computer-use agents, AI systems that execute software tasks by interacting with a computer, have typically been evaluated in ways that conflate task difficulty with the interaction modality used. A new paper from researchers introduces a benchmark designed to untangle those factors. It compares screen-only GUI agents and skill-mediated CLI agents on identical goals, initial states, and verifiers across 440 desktop tasks spanning 18 applications and 12 workflow categories.

Controlled comparison of interaction modalities

The core insight of the study is that existing evaluations blur together interaction modality with differences in tasks, initial states, verifiers, and permitted actions. To fix that, the researchers built a matched execution-layer benchmark. Both types of agents receive identical goals, states, and final-state verifiers, but each is restricted to modality-native actions: GUI agents can only see and click the screen; CLI agents can only use programmatic command interfaces via predefined skills.

The benchmark covers 440 tasks across common desktop applications in categories such as email management, spreadsheet operations, file organization, web browsing, and code editing. Each task is designed to be equally achievable via both interaction modalities, given sufficient capability.

Results: GUI outpaces baseline CLI

The strongest GUI agent tested achieved a 59.1% full pass rate, beating the strongest original-skill CLI agent at 48.2%. That 10.9 percentage point gap might suggest that GUI interaction is inherently more suitable for desktop tasks. But the researchers dug deeper to understand why CLI agents were falling short.

“The CLI agents we tested were limited by their skill libraries, the set of programmatic commands they could use to interact with software,” the paper notes. “When we augmented those skills using the verifier that checks final state, CLI success jumped to 69.3%, substantially outperforming the GUI agents.” This finding indicates that much of the CLI deficit comes from incomplete skill coverage rather than the model's underlying reasoning ability.

Different bottlenecks for different modalities

The study reveals distinct execution bottlenecks for each modality. GUI agents are limited by reliable grounded interaction over long-horizon workflows. They struggle with tasks requiring many sequential steps, where small perception errors compound. CLI agents, while more reliable per action, are limited by the coverage and scalability of their skill interfaces. Without proper skill coverage, they simply cannot attempt certain actions.

“GUI agents must learn to click the right pixel, scroll the right amount, and read screen text accurately over dozens of steps,” the researchers write. “CLI agents can execute commands precisely but only if they have a skill for that particular action. The bottleneck shifts from perceptual grounding to skill completeness.”

Implications for AI agent design

These results have practical implications for developers building computer-use agents. Skill augmentation, automatically generating or retrieving skills based on task requirements, emerges as a powerful technique to boost CLI agent performance. The verifier-guided augmentation approach used in the study demonstrates that even modest skill expansion can yield significant gains.

For GUI agents, the path to improvement lies in better visual grounding, longer-context video understanding, and more robust error recovery strategies. Hybrid approaches that combine GUI perception with CLI execution when available could potentially offer the best of both worlds.

Looking forward

The benchmark itself, comprising 440 tasks with matched GUI and CLI interfaces, initial states, and verifiers, has been publicly released to facilitate further research. The paper suggests that future work should explore adaptive skill retrieval, multimodal fusion of screen information with command execution, and tasks requiring real-time collaboration between agents operating in different modalities.

As computer-use agents move from research labs to production systems, understanding the trade-offs between interaction modalities becomes critical. This study provides a controlled, evidence-based framework for making those design decisions.