AI agents
3 published articles
AI Agents
AI agents can't finish a Java migration without the build server telling them to stop
IBM Research introduces ScarfBench, an open benchmark for evaluating AI agents on enterprise Java framework migration. Early tests reveal that frontier agents are systematically overconfident about their own results, that configuration layers dominate effort, and that environment issues like Docker caches regularly derail migrations even when code transformations succeed.
2026-07-06
Open source
Microsoft just opened a codebase that kills the worst part of enterprise data work
Data Formulator 0.7 is an open-source AI system from Microsoft Research that helps enterprise teams connect fragmented data sources, prepare and analyze data with context-aware agents, and iteratively refine visualizations in a shared workspace, all without requiring SQL or programming skills.
2026-07-03
Benchmark
Oxford's GauntletBench puts AI agents through 100 real tasks. They failed 81% of them.
Oxford's GauntletBench puts AI agents through 100 challenging real-world tasks. Frontier systems cap out at 19.1% success, far below human performance. The benchmark targets overlooked capabilities in temporal perception, graphical understanding, and 3D reasoning across five professional applications.
2026-07-01