Aimip phase 1: A new benchmark to test ai climate models

A new crop of AI models can simulate Earth's climate far more efficiently than traditional systems. But the field has lacked rigorous, shared ways to test whether those models are actually accurate and reliable. That's the gap AIMIP, the AI Model Intercomparison Project, aims to close. It's a community effort designed to support scientific understanding and open evaluation of AI models for climate forecasting.

AIMIP brings together multiple modeling groups, including NVIDIA, Google Research, and others, around a shared benchmark experiment and dataset. The idea is straightforward: give everyone the same test, then compare results on common outputs and evaluation criteria. That makes it easier to build confidence in how these models are assessed.

With Phase 1 now complete, the team has released a dataset of AI weather and climate model forecasts for the benchmark experiment, along with a report and evaluations. The findings show that AI models are competitive on key climate metrics but still have trouble in some areas.

Leveraging a revolution in weather and climate forecasts

AI climate models are a relatively new phenomenon, but they build on years of rapid progress in using AI to predict short-term weather patterns. Like their weather-forecasting cousins, they rely on the ERA5 dataset of historical observations spanning the entire atmosphere. These AI-driven forecasts now regularly beat conventional weather models at key skill metrics for forecasts 1-10 days out, as demonstrated on WeatherBench. And they do it with extraordinary speed, using far less computational power.

The leap to climate modeling, simulating the atmosphere over decades or centuries, comes with its own set of challenges. Until recently, there were few AI models that could handle climate timescales in a way that resembles a traditional model. And unlike WeatherBench, the benchmarks and metrics for evaluating these models are far from obvious.

Climate models and the MIPs

Physically based climate models have been around for decades. They simulate Earth's climate under specific scenarios over periods of decades or centuries by using physical laws to predict the weather on short timescales, repeating the process for the entire globe. From that, they produce averages and extremes, average temperature and precipitation for a given location, say, or the likelihood of a heat wave or tropical storm.

These models also have to account for changes in the ocean and sea ice over time, because on long timescales those factors meaningfully affect the weather. And they must evaluate a range of possible hazards and scenarios, such as rising greenhouse gas (GHG) emissions.

To evaluate climate models, the scientific community relies on model intercomparison projects, or MIPs. A MIP is a standardized experiment that climate models must execute, providing common outputs for evaluation. The ongoing Coupled Model Intercomparison Project, or CMIP, has been the driving force behind efforts to develop accurate model forecasts of the effects of GHG emissions.

AI climate modeling offers the same promise as AI weather forecasting: forecasts made with revolutionary speed and efficiency, using up to three orders of magnitude less compute than physically based models. That could unlock scientific discovery for a much wider range of users. But only in the last two years or so have AI models from multiple groups, using a variety of architectures, demonstrated they can make stable, high-fidelity predictions for decades and centuries. Whether they can correctly respond to different climate scenarios is still largely unknown.

AIMIP Phase 1: Specification and submissions

AIMIP Phase 1 is the project's first shared benchmark experiment. It's designed to compare AI climate models under a common setup while keeping the scope narrow enough for broad participation. Models must forecast the state of the global atmosphere over 1979-2024, with monthly and daily output frequencies. They must be trained only on ERA5 historical observations from 1979-2014, leaving the last decade as test data. The choice of AI architecture is up to the participants.

Ocean and sea ice states are prescribed with historically observed values. At this early stage, the goal is to focus on the behavior of the atmosphere alone. In future AIMIP phases, it may become possible for AI to simulate the ocean, sea ice, and other Earth system components via a coupled climate model.

In Phase 1, models must output temperature, humidity, and winds at seven levels in the atmosphere, as well as surface temperature, precipitation, and other key weather variables. They must also make their outputs compatible with typical CMIP format specifications to facilitate comparison with conventional models and evaluation tools.

Eight model simulations were submitted by Ai2 Climate Modeling and five outside groups: the ArchesWeather team, NVIDIA, the University of Washington, the University of Maryland, and Google Research.

Faithful representation of the historical climate, but challenges in predicting its changes

The team evaluated how well AI climate models simulate the historical climate and its changes over the past several decades. They found that AI models, almost regardless of architectural choices, do very well at simulating average historical climate patterns, typically beating a conventional physically based model. The most accurate AI models reduce the time-averaged error in fields like near-surface air temperature by a factor of 2.

A more demanding test is whether the models capture the long-term warming trend visible in the historical record, especially beyond their training period and into the held-out final decade of ERA5 data. The picture there is more mixed. Some models track the warming trend quite well, while others underestimate it significantly. Generalizing to future conditions is essential for climate change projections, though it may be less critical for other uses, such as informatics or sampling climate risk factors during an AI model's training period.

Researchers also evaluated the models' ability to simulate atmospheric responses to El Niño ocean conditions, day-to-day atmospheric variability, and a true out-of-sample shock: an instantaneous 2 or 4 degree Celsius warming of the global ocean surface. That scenario isn't physically likely, but it's useful for understanding how AI models might generalize to unseen conditions. Not surprisingly, the models' predictions diverge significantly in this out-of-sample case, with some producing what appear to be physically implausible results.

Going forward: Open dataset and community evaluations

The AIMIP Phase 1 dataset is being hosted through the German Climate Computing Center (DKRZ), with publication to the Earth System Grid Federation (ESGF) planned to make it broadly accessible to the climate science community. Scientists are already using the dataset to carry out further evaluations, with the team's work serving as an entry point for continued research.

The results from Phase 1 suggest that one of the central challenges for AI climate models is responding robustly to a range of climate scenarios. Generalization will be critical if these models are to be widely adopted. In particular, researchers need to be able to trust how AI climate models behave under unseen GHG emissions scenarios. Conventional climate model outputs may provide training data for some of those cases, but additional AI-specific approaches will likely be needed.

If Phase 1 proves valuable, and if AI climate modeling continues to advance at its current pace, future AIMIP phases will follow. Those would likely expand to more complex coupled modeling, including ocean and sea ice dynamics, along with a broader set of scenarios such as GHG emissions pathways, and more extensive output requirements and evaluations.