AI agents can't finish a Java migration without the build server telling them to stop

Framework migration is the kind of task that sounds like a perfect fit for an AI coding agent: mechanical, repetitive, governed by well-documented rules. Replace a Spring annotation with a Jakarta equivalent, adjust a persistence descriptor, update a Maven dependency. A large language model should handle this in its sleep.

It turns out it cannot, at least not reliably, and certainly not without significant human oversight.

IBM Research today released ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark designed to measure whether AI agents can actually migrate real enterprise Java applications between frameworks such as Spring, Jakarta EE, and Quarkus. The headline finding: frontier agents compile code about 70 percent of the time, but only about 50 percent of migrated applications actually deploy and pass behavioral validation. The agents themselves consistently overestimate their own success.

What makes framework migration fundamentally different

ScarfBench was built because existing software engineering benchmarks, which test bug fixing or isolated code generation, do not capture the structural intricacy of a cross-framework migration. Portfolio-level Java modernization involves changing dependency injection, persistence configuration, query syntax, build descriptors, and runtime packaging. An error in any one layer can prevent deployment, and failures cascade.

The benchmark includes both focused migration tasks, single-component translations, and whole-application migrations drawn from a JSR-based enterprise Java taxonomy. Each application is evaluated on three criteria: does it build, does it deploy, and does it preserve behavior? That third criterion is the one most existing benchmarks skip.

Agents are overconfident, and that is a real risk

Perhaps the most concerning finding in the paper is the gap between agent-reported success and independently verified success. Claude Code reported successful builds for 29 out of 30 whole-application migrations. Only 22 of those applications actually built. One application the agent classified as a failure turned out to build correctly.

“Agent self-assessment should not be treated as a reliable signal of migration completion,” the authors write. In a real-world enterprise setting, false confidence could lead teams to ship broken applications, or worse, to skip automated validation entirely on the assumption that the agent has already verified its work. ScarfBench makes the case that independent build and test validation is not optional, it is the only signal that matters.

Configuration is where migrations go to die

When researchers tracked which layers of an application agents revisited most frequently during a migration, the pattern was clear. Configuration files, application.properties, pom.xml, beans.xml, dominated the iteration log. The most common transition in every migration was between configuration and the web layer, followed by service-to-database transitions.

This is not a linear process. Agents do not translate source files first and then fix configuration. They ping-pong between configuration, web, database, and service layers, repeatedly adjusting dependencies as framework incompatibilities emerge. The migration is an iterative dependency-resolution exercise, not a source-to-source transformation.

Docker caches, port collisions, and Maven wrappers

Not all failures are about code. The evaluation team catalogued a range of environmental issues that derailed validations: Docker cache inconsistencies, port connectivity problems, Maven wrapper version mismatches, and build tooling quirks. These operational failures often delayed testing long after the agent had finished what it thought was the code migration.

In a production setting, a human operator would spot a port collision in seconds. An agent, unless explicitly prompted to check infrastructure, simply reports failure and may never isolate the root cause.

Target framework matters

Migration difficulty varies significantly depending on the destination framework. Jakarta EE proved the hardest target, a finding the researchers attribute to its heavier reliance on XML descriptors and its more complex deployment model compared to Spring Boot or Quarkus. Quarkus migrations fell somewhere in between, with build success rates notably higher than behavioral success rates, suggesting that Quarkus compiles easily but its reactive runtime semantics trip up agents during deployment.

The benchmark is designed to be extensible, and the team invites contributions of new migration scenarios. The dataset, evaluation infrastructure, and leaderboard are all open-source.

ScarfBench will not be the last word on AI-assisted migration, but it provides the first reliable measuring stick. The gap between compile success and behavioral success is the gap between an agent that looks productive and one that actually is.