Google DeepMind's Gemma 4 turns 26 billion parameters into a reasoning machine that fits on one GPU

The line between proprietary frontier models and open-weight releases has blurred again. Google DeepMind published the Gemma 4 technical report on July 27, 2025, detailing a family of models that challenge the assumption that open-weight models must lag behind closed systems in reasoning, multi-modal understanding, and efficiency.

Gemma 4 is not a single model. It is three configurations: a 2.6-billion-parameter dense model, a 9.2-billion-parameter dense model, and a 26-billion-parameter mixture-of-experts model that activates roughly 9.2 billion parameters per token. The MoE variant, built on a decoder-only transformer augmented with Vision Transformer encoders for image understanding, is the standout. It handles text, images, and short videos natively, and its 1 million-token context window puts it alongside the longest-context models in the open-weight ecosystem.

A deliberate architectural split

Many labs have converged on dense transformers for language and separate vision models. Gemma 4 fuses both modalities into a single trainable stack. The text backbone uses a causal transformer with Grouped-Query Attention, Root Mean Square Normalization, and Rotary Position Embeddings. The vision side reuses the same transformer layers after a ViT patch-encoder, with cross-attention at earlier layers and a final projector. This design allows the model to reason jointly over language and visual inputs without the overhead of separate encoders at inference time.

The MoE router is trained with a softmax top-2 gating mechanism, a choice that prioritizes load balancing across experts. During training, the router distributes tokens evenly enough that no auxiliary loss is required for balancing, a detail that reduces training complexity.

Context length that changes use cases

The 1 million-token context window on the MoE model is not a theoretical maximum. The report benchmarks retrieval at the 1 million-token range using the RULER and LongContext benchmarks, showing that the model maintains near-perfect retrieval accuracy. For developers building applications that require reasoning over entire codebases, long legal documents, or multi-hour video transcripts, this context length eliminates the need for chunking and retrieval pipelines that add latency and complexity.

The smaller dense models support 256,000-token contexts, still well above average for their size class.

Performance that reshapes expectations for open weights

The reported benchmarks are where Gemma 4 invites direct comparison with much larger models. On the MMLU-Pro reasoning benchmark, the MoE variant scores 75.3%, outperforming closed models like GPT-4o (73.6%) and approaching o3-mini (80.7%). On LiveCodeBench (v6), it achieves 48.2%, again surpassing GPT-4o and trailing o3-mini and Claude 4 Sonnet. On GPQA-Diamond, a graduate-level science reasoning benchmark, the MoE model scores 74.9%, beating all non-Gemini models tested.

These numbers are significant because they come from a 26B-parameter MoE model that activates only about 9B parameters per token. The parameter-efficiency story is clear: Google DeepMind has squeezed reasoning capability from a smaller active parameter count, translating to lower inference costs and faster generation on consumer hardware.

The reasoning improvement pipeline

One of the report's more technical sections details a reinforcement-learning-based reasoning improvement method the team calls Round. The approach uses a sparse reward model and iterative fine-tuning to push the model toward longer chains of thought without collapsing into superficial reasoning. Ablation studies show that Round improves GSM8K math scores by 12.4 percentage points and HumanEval coding scores by 8.9 points compared to the base supervised fine-tuned model.

This matters because reasoning improvement techniques have historically been the domain of proprietary labs with massive compute budgets. By documenting and open-sourcing the method alongside the model weights, Google DeepMind hands the open-source community a known working procedure for improving reasoning in MoE transformers.

Safety and red-teaming infrastructure

The report dedicates substantial space to safety evaluations. The team conducted red-teaming with 150 internal testers and 50 external domain experts, covering areas from chemical and biological weapons to cyber attack capabilities. The models were tested on the Humanity's Last Exam safety benchmark, and the report includes a taxonomy of risks and mitigations. Notably, the Gemma 4 models were found to have "negligible capability" for assisting in the creation of CBRNE weapons and no increase in offensive cyber capabilities compared to earlier Gemma versions.

Google DeepMind also released a new safety dataset called ShieldGemma 2, trained on Gemma 4 outputs, that scores potentially harmful content across all three models. All model variants scored within the "low risk" range on the Google Frontier Safety Framework's tiered evaluation.

What this means for the ecosystem

Gemma 4 shifts the competitive landscape for open-weight models. It arrives at a moment when labs like Meta with Llama 4, Alibaba with Qwen 3, and Mistral AI have all pushed open models toward larger sizes and better reasoning. Gemma 4's differentiator is not raw size but architectural efficiency and multi-modal integration from the start.

For developers, the implications are practical: a model that runs on a single consumer GPU in some configurations, supports 1 million tokens of context, reasons competitively with closed APIs, and is distributed under a permissive license (CC BY 4.0 for the report, with model weights under the Gemma Terms of Use).

Google DeepMind has not announced a partnership ecosystem or enterprise support tier for Gemma 4, as it did with earlier Gemma versions. The report reads more like a research publication than a product launch. Detailed, thorough, and leaving deployment to the community.