Gemma 4 runs 90% faster in Ollama 0.31 with a trick that needs no config

Ollama 0.31 gives Gemma 4 a serious speed boost on Apple Silicon. Token generation can run up to 90 percent faster on a coding-agent benchmark. The trick is multi-token prediction (MTP). A small draft model proposes several tokens at once, and the main model verifies the whole batch in a single pass. No configuration needed. It just works out of the box.

That matters most for coding agents, where the model gets called over and over while reading files, running tools, and solving tasks. Faster generation makes those agents feel noticeably snappier. The benchmark numbers come from realistic conditions, not cherry-picked demos. For a deeper look at how coding agents fail under real workloads, see UniClawBench's findings on agent behavior.

How multi-token prediction works in Ollama

Gemma 4 ships with a lightweight draft model that runs alongside the main model and proposes the next several tokens. The main model then verifies the entire proposal in one pass, keeping any tokens it agrees with. The draft model is a fraction of the main model's size, so its proposals cost almost nothing to compute. When the proposals are right, the model commits multiple tokens for the price of one.

Code is especially predictable: closing brackets, repeated identifiers, boilerplate. The draft model's proposals get accepted often. That is why MTP excels on coding benchmarks like the Aider polyglot benchmark, where a real agent works through a series of programming tasks. Context matters here: the hardest problem in AI coding agents isn't code, it's counting the hours saved versus human effort.

Ollama auto-tunes the draft length at runtime. It tracks how often proposals are accepted and how long each verification pass takes. Then it picks the length that yields the highest tokens per second and adjusts as the text changes. If proposals stop being accepted, the engine falls back to one-token-at-a-time decoding, so MTP never drags generation down.

Speculative decoding and the MLX kernel contribution

Each round starts with the draft model predicting a token, feeding it back in to predict the next, and repeating until it has a short run of proposals, typically 2 to 8 tokens. The main model then verifies the whole run at once, sampling at each position to decide which proposals are accepted. The entire process stays on the GPU as a single pass: drafting, sampling, verification, and final sampling, with no return to the CPU in between.

If a proposal gets rejected, the engine rewinds the key-value cache to the last accepted token. That is cheap because it only touches the latest entries. The system records a rollback point before each proposal, so rejection never redoes earlier work.

The expensive part of MTP is verification, not drafting. Verification runs the full model over the entire batch of proposals at once, and that batch size (2 to 8 tokens) falls awkwardly between the typical decode (one token) and prefill (large batch) sizes. To solve that, the Ollama team contributed a custom kernel to MLX that reads and unpacks each block of weights once and reuses it across the whole batch. No redundant memory reads. On an M5 Max with nvfp4 quantization, it makes Gemma 4's largest matrix multiplications 2x to 2.5x faster. Same computation, less memory traffic. This type of infrastructure bet is exactly what Ollama's $88 million push for open-weight models is all about.

Getting started

You can download Ollama 0.31 or later for macOS and launch a coding agent with Gemma 4 using a single command. If you already downloaded Gemma 4, re-pull the model to get the MTP-enabled version. Ollama launch also works with Codex, Droid, OpenCode, Copilot, and other agent frameworks.

Gemma 4 is the first model to get this performance improvement. Ollama says more models will get MTP support in future releases. For a broader view of what this means for the open-weight model race, check how Gemma 4 rewrote the rules for open-weight models.