The one metric AI coding agents can't fake: human hours

Six months ago, every CTO worried their team was not consuming enough AI tokens. That worry has flipped into a different anxiety: how to tell which tokens actually delivered value and which were wasted on looping, hallucination, or directionless prompting. For an organization running thousands of autonomous agent sessions a month, the problem scales beyond what any engineering manager can manually triage.The subtle trap waiting for AI agents in production

Cognition Labs, the company behind Devin, the so-called autonomous AI software engineer, published a detailed account of how it built a system to solve exactly this problem. In a research post released Monday, the company described an automated estimator that reviews each completed Devin session, classifies whether it produced useful output, and then estimates how long a human engineer would have taken to produce the same work. The system is already running in production with paying customers.Cognition's new coding agent scores near frontier…

The company claims this is the first automated system measuring AI engineering productivity in production.

Why hours, not dollars

The choice of metric was deliberate. Measuring dollar impact directly, revenue from a shipped feature, cost avoided by a squashed bug, remains an unsolved problem for the entire software engineering industry. Even human engineers, filling out their own sprint retrospectives, struggle to quantify the business value of a pull request.The hidden tax on vibe-coded projects that shows up…

Raw activity metrics like lines of code, commits, or PRs are easy to collect but notoriously misleading. A mechanical refactor can touch thousands of lines in an afternoon; a two-line bug fix can represent hours of investigation. Many valuable tasks, triaging bugs, running analytics queries, reviewing code, produce no code at all.

The middle ground Cognition selected is human engineering hours: how long would a competent human engineer have taken to produce the same output? Hours are how organizations already denominate engineering value, from salaries to contractor rates to time-to-market calculations. The metric is standardized across organizations and convertible to dollars via engineering rates.

But not all hours are equal. If a session's pull requests were all subsequently closed without merging, that session likely was not valuable. Cognition therefore built a classifier to filter out unproductive sessions before running the hour estimator.

Building ground truth from user reports

To train and validate the estimator, Cognition collected a ground-truth dataset by asking Devin users, across a diverse set of enterprise customers, to review recent representative sessions and estimate how long each would have taken without Devin. The data came from live interviews and a survey.Anthropic and DXC Technology Launch Global Alliance to…

The richness of the session traces turned out to be critical. Every Devin session produces a full execution trace: the user's request, every action the agent took, the resulting code, and codebase context. The company noted this gives them a record of production engineering work at a level of detail that is difficult to obtain from surveys, aggregate activity metrics, or open-source benchmarks alone.

The unproductive-session classifier removes roughly 30% of sessions, depending on the customer. Sessions where Devin lacked access to carry out the task, asked for clarification but never received an answer, or was otherwise unable to meaningfully advance are discarded. Sessions that produce merged PRs are retained; sessions without PRs are kept only if the classifier deems them genuinely productive, tasks like finding unused dependencies, scanning for security vulnerabilities, or reviewing a pull request.

The estimator: reasoning about human paths, not agent trajectories

The estimator itself is an LLM-powered agent that receives two key inputs: the full session context (user messages, PR output, agent trace, codebase context from DeepWiki) and a carefully engineered prompt.

The prompt design is where the real engineering judgment matters. Cognition identified several principles to avoid overestimating output:

• Reason about the human's path, not the agent's. Agents take detours, they recover from environment failures, produce artifacts a solo engineer would not, and retry failed actions. The estimator discounts all of that.
• Credit only work the user did not already specify. If a user provides a bug report with no proposed fix, the estimator includes triage time. If the user provides a full implementation plan, only the implementation time is counted.
• Account for codebase familiarity. Users told Cognition that one of Devin's clearest advantages is working in unfamiliar or legacy codebases that would cost a human developer a day of ramp-up. The estimator infers the required familiarity from the session and includes exploration time when appropriate. When there is no signal, it assumes typical familiarity, an engineer who knows the high-level architecture but has not memorized every function.
• Assume relevant expertise. A recurring interview theme was that Devin lets people do things they could not do on their own, a backend engineer shipping frontend and data-science work. Cognition conservatively assumes the reference engineer already has the required expertise, which understates effort in many cases where a human would have needed to learn a new language or framework.

Validation results: R² 0.70, deliberately conservative

On the held-out evaluation set, the estimator achieves an R² of 0.70 and a Spearman correlation of 0.85 (p < 0.001). About 60% of sessions fall within a factor of 2 of the true estimate. Individual estimates are noisy, 2-3x errors in either direction are common, but because errors are roughly unbiased and independent, they cancel as session count grows, and the aggregate converges toward the human-reported total.

Cognition found that roughly half the residual disagreement lies between users, not within a user's own sessions. The company considered per-user calibration but decided against it for simplicity, opting instead to estimate relative to an average user.

A notable finding: the initial, uncalibrated model consistently underestimated. Cognition corrected this by fitting a linear regression in log-space, which produced a nearly constant multiplier. Even after calibration, however, the sum of the model's hour estimates remains lower than the sum of human estimates. Cognition acknowledges this gap is expected, an unbiased estimator in log-space underestimates when summed in linear space, and reports the unadjusted figure as a deliberately conservative underestimate.

Code volume is a poor proxy

The company also tested simpler predictors to understand how much signal comes from the final code change versus the full session trace. A regression using total lines changed (additions plus deletions) against human estimates performed poorly, with an R² of 0.13, confirming that code volume is a weak proxy for engineering effort. An estimator given only the agent's edit tool calls, with no user messages or session activity, performed better but still lagged the full estimator.

The company concluded that the effort in a session often comes from investigation, diagnosis, environment setup, reasoning through tradeoffs, or producing useful non-code outputs. Those signals are visible in the session trace but not always in the final code change.

Comparison with prior work

Two recent studies have used LLMs for effort estimation. METR (2026) used GPT-4o and GPT-5 to estimate human-equivalent times from compressed Claude Code transcripts, achieving an R² of 0.82, higher than Cognition's 0.70, but on a more homogeneous dataset collected from METR technical staff rather than diverse enterprise users.

Anthropic (2026) estimated task duration on open-source Jira tickets using only the ticket title and description, reaching an R² of 0.34. Cognition's system achieves R² 0.70 versus Anthropic's 0.34, demonstrating that granular session data significantly improves accuracy.Anthropic Launches Claude Sonnet 5: A New Frontier in…

Caveats and threats to validity

Cognition is transparent about the limitations. The ground truth is self-reported by users who were aware they were talking to Cognition, which may introduce bias. Users who voluntarily responded may skew toward the most engaged. And hours saved are not the same as business value, one hour fixing a critical production bug differs dramatically from one hour on a project later canceled.Cognition's new coding agent scores near frontier…

The company noted that they measure engineering capacity, not whether that capacity was deployed on high-value work. The system also does not account for quality: if Devin introduces a subtle bug detected after merging, its uplift on that task is negative. The merged-PR filter removes the clearest failures but not post-merge defects.

Despite these caveats, the system represents a significant step toward making autonomous AI coding agents auditable in terms that engineering leaders already understand: hours, dollars, and ROI. The question is whether the industry will converge on a standard for measuring that ROI, and who will set it.Microsoft's bet on small models for agentic AI is about…