Benchmark Deep Dive
ARC-AGI-2: The Benchmark That Measures Fluid Intelligence in AI Systems
ARC-AGI-2 tests AI systems on fluid intelligence through visual grid puzzles that can't be solved by memorization. Top frontier models now score 75-85%, but the grand prize of $700,000 remains unclaimed. Here's a deep dive into the benchmark's design, scoring, and current leaderboard.
The Abstraction and Reasoning Corpus, now in its second generation, is no ordinary benchmark. Unlike MMLU or GPQA Diamond, which reward deep knowledge retrieval, ARC-AGI-2 is engineered to isolate one specific cognitive capability: fluid intelligence. The term, drawn from Raymond Cattell's psychology framework, refers to the on-the-fly ability to reason about novel problems and identify patterns under uncertainty. Crystallized intelligence, by contrast, is the accumulated body of facts a person, or model, has internalized over time. Most LLM benchmarks reward the latter; ARC-AGI-2 is built to reward the former.
How ARC-AGI-2 Works
Each task presents a handful of input/output grid pairs, typically two to four, and a new test input. The model must infer the transformation rule that maps each input to its output and then apply it to the test grid. Cells are integers from 0 to 9, mapped to a fixed ten-color palette that is purely decorative. The difficulty lives in the structure, not the colors.
The format is deliberately few-shot. Two or three examples are rarely enough to disambiguate a rule by pattern matching alone. To produce the correct output, the model must generate hypotheses, check each against the demonstration pairs, refine when a hypothesis fails, and commit to the rule that survives every example. The rules are not drawn from any published taxonomy; each puzzle is a one-off problem.
Why Visual Grids?
The choice of visual grids is central to the benchmark's design. Knowledge benchmarks like MMLU and GPQA Diamond reward the ability to retrieve facts seen many times during training. ARC-AGI-2 deliberately avoids that. A grid transformation rule that has never appeared in a textbook gives the benchmark room to discriminate genuine pattern induction from memorized solutions. The capability it targets, fluid intelligence, is harder to fake because pre-training offers no advantage on a problem never encountered before.
Human Baseline and Prize Threshold
Two human baseline numbers matter here. The ARC-AGI-2 calibration study, run during the 2025 redesign, recruited hundreds of human participants. Every task in the eval set was solved by at least two humans in two attempts or less, a 100% panel-completion rate. Average individual performance, reported in the GitHub repository, landed at 66%. The grand prize threshold is set higher: greater than 85% on the private holdout, within Kaggle's compute and runtime efficiency limits, to claim the ARC Prize 2026 grand prize of $700,000. That bar has not been reached in either the 2024 or 2025 competition years.
Current Leaderboard (April 2026)
According to BenchLM's tracked benchmarks data, the top ten models on ARC-AGI-2 are:
- GPT-5.5, 85 (Reasoning)
- GPT-5.4 Pro, 83.3 (Reasoning)
- Gemini 3.1 Pro, 77.1 (Non-Reasoning)
- Claude Opus 4.7 (Adaptive), 75.8 (Reasoning)
- GPT-5.4, 73.3 (Reasoning)
- Claude Opus 4.6, 68.8 (Non-Reasoning)
- Claude Sonnet 4.6, 59 (Non-Reasoning)
- GPT-5.2 Pro, 54.2 (Reasoning)
- Grok 4.20, 53.3 (Reasoning)
- GPT-5.2, 52.9 (Reasoning)
The spread is significant: top model at 85, tenth at 52.9, with models like DeepSeek V3.2 at 4 and o3 at 3. ARC-AGI-2 does not compress at either end, rare for a benchmark with a 66% human average. It provides genuine separation across the frontier of reasoning model capability.
Why Reasoning Models Excel
Of the top five, four use explicit reasoning configurations. The mechanism is intuitive: ARC puzzles benefit from iterative hypothesis-testing, generating a candidate rule, checking it against demonstration pairs, refining when it fails, and committing to a rule that survives every example. That loop is exactly what chain-of-thought scaffolding enables. The cost is longer latency and higher per-task spend, but the performance gain is substantial.
What ARC-AGI-2 Does and Does Not Measure
ARC-AGI-2 correlates with agentic task performance on novel domains, ambiguous coding tasks, and reasoning problems that cannot be solved by retrieval. It does not predict knowledge recall (MMLU, GPQA), instruction-following fidelity (IFEval), conversational quality (Arena Elo), or multilingual reasoning (MGSM). A model with 75 on ARC-AGI-2 and 60 on HLE is fluid-intelligence strong but knowledge-thin; the inverse is also possible. No single test covers all capabilities.
The 'AGI' in the name reflects François Chollet's bet that fluid intelligence is the missing piece for general AI. Whether the bet is right remains an open empirical question, but ARC-AGI-2 is designed to measure progress toward that goal with unusual rigor.