ProgramBench: every public AI scores 0% on the hardest coding test yet

ProgramBench is a tough new LLM coding benchmark that targets a capability most leaderboards barely touch: can a model reconstruct an entire program from observed behavior, without ever seeing the source code?

The setup is straightforward but punishing. An agent receives a compiled executable and its usage documentation. Its job: probe the program, infer how it works, pick an implementation strategy, write source code, craft a build script, and submit a candidate program. The output is then compared against the original through hidden behavioral tests.

That is a far cry from completing a single function, solving a contest problem, or patching a known repository.

What ProgramBench Tests That Other Benchmarks Don't

ProgramBench matters most for evaluating coding agents rather than chat models. If your workflow is "ask the model to write a small function," ProgramBench is probably too hard and too indirect. HumanEval, LiveCodeBench, and language-specific coding evals will tell you more. If your workflow is "ask the model to patch a known repository," SWE-bench Pro, SWE-Rebench, and SWE-bench Verified remain more directly relevant.

ProgramBench becomes interesting when the job is less structured. For model buyers, it is not yet a "pick the top model and ship it" leaderboard. It is more of a warning: current coding agents still stumble when the source code, issue description, and scaffolding vanish. For agent builders, it is a diagnostic tool, it reveals whether the agent can investigate before it implements.

ProgramBench is built around the concept of cleanroom reconstruction. Each task starts with a compiled executable and documentation. The model must decide which questions to ask the executable. It can run the program with inputs and observe outputs, but it cannot peek at the underlying implementation. That makes ProgramBench a benchmark for architecture and specification discovery, not just code writing.

How ProgramBench Compares to Major Coding Benchmarks

Most coding benchmarks still hand models a lot of structure. HumanEval provides a function signature and docstring. SWE-bench Verified gives an existing repository and a specific issue. LiveCodeBench delivers competitive-programming-style problem statements. ProgramBench strips away most of that scaffolding. It asks whether the agent can reconstruct target behavior from interaction alone.

That distinction matters because real software engineering often starts from partial specifications. Developers reverse-engineer workflows, probe APIs, explore aging systems, and infer edge cases from observed behavior. ProgramBench compresses that kind of work into an evaluation setting.

The closest comparison is SWE-bench, but the task shape is almost inverted. SWE-bench asks whether the agent can modify a real codebase correctly. ProgramBench asks whether the agent can discover what a program does and build a new codebase that behaves identically.

Current Leaderboard: A Sobering Snapshot

The initial public leaderboard is deliberately sobering:

• Claude Opus 4.7: 0.0% fully resolved, 3.0% almost resolved
• Claude Opus 4.6: 0.0% fully resolved, 2.5% almost resolved
• Claude Sonnet 4.6: 0.0% fully resolved, 1.0% almost resolved
• GPT 5.4: 0.0% fully resolved, 0.0% almost resolved
• Gemini 3.1 Pro: 0.0% fully resolved, 0.0% almost resolved

The primary metric is tasks fully resolved. Every public model currently sits at 0%. ProgramBench also reports "almost resolved" tasks, runs that pass at least 95% of behavioral tests. BenchLM uses that auxiliary metric on the display page because it is the only visible separator between models right now.

How to Read the Leaderboard Correctly

The ProgramBench leaderboard demands a more careful reading than a normal coding benchmark. On most coding leaderboards, a higher score directly means more solved tasks. On ProgramBench today, the official primary metric is flat: every evaluated public model is at 0% fully resolved. The only separation comes from the auxiliary "almost resolved" rate.

That means three things. First, do not over-rank the current top three. Claude Opus 4.7 leading at 3.0% almost resolved is a meaningful signal that it got closer more often, but it is not a production-level success rate. Second, do not treat 0.0% almost resolved as identical model quality. A model can fail ProgramBench in many ways. Third, expect harness effects to matter, ProgramBench is an agent benchmark, not a pure model benchmark.

Failures Exposed: Why Scores Are Near Zero

Low ProgramBench scores expose several failure modes at once. Shallow probing leads agents to infer too much from too few observations. Premature implementation, encouraged by many agent workflows optimized for quick editing, hurts when the main challenge is discovering the spec. Missing negative cases mean agents overlook invalid flags, errors, and exit codes. Weak architecture without a skeleton leads to brittle implementations that cannot scale. Poor self-testing prevents agents from validating their replacements. Overconfidence is a serious practical problem: agents often declare completion and submit incomplete work, as noted in the LocalLLaMA launch discussion. Tool-loop fragility can kill even strong base models.

Those failure modes are precisely why ProgramBench is useful. They are also why a future public leaderboard should report more than one number. For serious analysis, we need to know whether failures came from bad probing, bad code, failed builds, time limits, missing edge cases, or bad stopping decisions.

How to Run ProgramBench

The team has published the GitHub repository, Hugging Face datasets, and Docker/inference materials. The basic local workflow is to install the package and evaluate a submission with programbench eval. Agents must preserve the benchmark setup: no internet access, no direct source lookup, and no decompilation shortcuts. For teams building coding agents, the practical value is less about chasing the current leaderboard and more about running controlled internal experiments.

Next Steps for the Benchmark

ProgramBench will become more useful if public submissions separate model quality from harness quality, if open-weight results get broader coverage, and if the primary resolved metric moves. BenchLM tracks ProgramBench on a dedicated benchmark page and keeps it display-only on the coding leaderboard until the signal becomes more stable. For now, it is best treated as a frontier warning light about the limits of current coding agents in cleanroom reconstruction.