Aleph Alpha builds a theoretical inference model to decode DeepSeek V3 performance from hardware primitives

Aleph Alpha has shared a new theoretical inference model that estimates throughput for large mixture-of-experts (MoE) models, with DeepSeek V3 as its primary test case. The work aims to give engineers a more intuitive grasp of the trade-offs between latency, throughput, and cost across different hardware configurations.

Modeling inference from hardware primitives

The model takes specific hardware parameters as inputs, GPU count, interconnect bandwidth, memory bandwidth, and compute capacity, and outputs estimated tokens per second for a given MoE model. Aleph Alpha's team built it to systematically explore how architectural and hardware decisions influence real-world inference performance.

DeepSeek V3, one of the most popular open-source large language models today, recently introduced significant inference-time optimizations that make it surprisingly efficient to serve despite its enormous size. The Aleph Alpha model captures these optimizations and translates them into performance figures that can be compared across setups.

Key trade-offs revealed

The report identifies three primary bottleneck regimes that shift depending on the ratio of GPU count to interconnect speed:

• Compute-bound scenarios: When all-to-all communication is fast enough, throughput is limited by the raw FLOPs of the GPUs.
• Memory-bound scenarios: When model weights or KV cache dominate capacity, memory bandwidth becomes the limiting factor.
• Communication-bound scenarios: When inter-GPU bandwidth is insufficient, the cost of expert routing and all-to-all communication becomes the primary bottleneck.

By varying GPU count and interconnect speed, Aleph Alpha demonstrates how the bottleneck shifts across these regimes, enabling hardware selection tailored to specific deployment goals.

Practical implications for practitioners

The work is aimed at engineers and architects designing inference infrastructure for large MoE models. It provides a way to estimate performance without running extensive benchmarks, accelerating hardware procurement and deployment decisions.

Aleph Alpha's report includes detailed data and charts showing how changing the number of GPUs or interconnect bandwidth affects throughput and latency. It also explores the impact of batch size, precision (FP16 vs. FP8), and the number of active experts per token on overall throughput.

Why DeepSeek V3 matters for MoE inference

DeepSeek V3 has drawn attention for its scale, hundreds of billions of parameters, combined with an efficient MoE architecture that activates only a fraction of parameters per token. That design introduces unique inference challenges, particularly around expert routing and all-to-all communication across GPUs. Aleph Alpha's model tackles these challenges directly, offering a quantitative lens to understand how the architecture behaves under different hardware constraints.

The full report is available for download from Aleph Alpha's blog, with additional insights on how to apply the model to other MoE architectures under development.