mimalloc, Microsoft's tiny memory workhorse, is quietly powering AI at scale

In the competitive world of memory allocators, a project born inside a Microsoft Research lab has quietly become a workhorse, not just for small programming language runtimes, but for massive cloud services. The mimalloc (Microsoft malloc) allocator, an open-source drop-in replacement for malloc and free, is proving especially effective for the highly concurrent, memory-hungry workloads that define modern computing, including large language models (LLMs).

Developed by the RiSE group at Microsoft Research (MSR), mimalloc first appeared in 2020 as a fast allocator for the Lean theorem prover and the Koka programming language, both of which use novel compiler-guided reference counting. But its scalable design, the researchers wrote in a recent blog post, has worked exceedingly well for large services at Microsoft, leading to big improvements in response times for services like Bing, thanks to close cooperation with product teams.

Today, mimalloc is widely used both inside and outside the company. It powers NoGIL CPython 3.13+, is integrated into Unreal Engine and games like Death Stranding, and lives on GitHub with over 12,000 stars. Its Rust wrapper alone sees more than 100,000 downloads per day.

Compact Codebase, Strong Invariants

Despite its wide range, from small compilers to services with memory footprints topping 500 GiB and hundreds of threads, mimalloc's codebase stays compact at around 12,000 lines of C. The project emphasizes clear internal data structures with strong invariants, making it easier to understand and reason about than many industry allocators. That clarity has helped port it to Windows, macOS, Linux, FreeBSD, NetBSD, DragonFly, and multiple gaming consoles.

“As Fred Brooks already remarked in his famous book The Mythical Man-Month: ‘Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t need your flowchart; it’ll be obvious,’” the researchers wrote, explaining their design philosophy.

The transparent data structures also let Sam Gross and others adopt mimalloc as the concurrent allocator for NoGIL CPython, making it straightforward to implement cyclic garbage collection on top of the allocator.

Fast Path with Minimal Synchronization

At its heart, mimalloc uses thread-local heaps (theaps), much like other scalable allocators such as tcmalloc and jemalloc. Each thread gets its own heap, which owns a set of mimalloc pages (usually 64 KiB). Each page contains blocks of a fixed size, sorted into size classes to cut down on internal fragmentation. By giving each thread its own heap and pages, allocation and deallocation typically need no synchronization.

Atomic operations only come into play when a thread frees a block allocated by another thread. In practice, most allocations are tiny, often under 1 KiB, and for those, mimalloc provides an incredibly fast path. The main allocation function boils down to just a few x64 instructions with only two uncommon branches.

Similarly, mimalloc has a fast path for freeing blocks. Most blocks are freed by the thread that allocated them, and the allocator checks whether the current thread ID matches the ID stored in the corresponding page. If so, it pushes the block onto the page's free list without locks or atomic operations.

Three Free Lists and Randomized Inspiration

A key insight in mimalloc's design is its three free lists per 64 KiB page: one for allocations, one for freed blocks, and one for cross-thread frees. This means a program can easily have thousands of free lists, essential for scalability and cache locality.

The researchers took inspiration from randomized algorithms. “Many multi-threaded allocators rely on sophisticated concurrent data structures to synchronize access to shared free lists. In contrast, mimalloc uses a per-page thread-free list, where any thread can push a block using a simple atomic compare-and-swap,” they wrote. “Because there are thousands of such lists, the probability that multiple threads concurrently free blocks to the same page is low. As a result, most push operations are uncontended atomic updates.”

Balancing Scalability and Memory Efficiency

The design tackles a fundamental tension: giving each thread exclusive ownership minimizes synchronization but can waste memory. At the other extreme, sharing all pages globally with a single lock optimizes memory use but kills scalability.

Benchmark results illustrate the trade-off. The standard system allocator showed excellent memory efficiency (1.1x committed versus live data) but allocated only 56 GiB over the benchmark period. Another highly concurrent allocator allocated 262 GiB but committed four times more memory than the live data, a ratio that becomes unacceptable at large footprints.

“The final graph shows the most recent mimalloc allocator. Like the second allocator, it allocates 262 GiB over the benchmark duration, while reducing committed memory to 1.3x the live data, which achieves scalability and efficient memory sharing between threads,” the researchers noted.

That improvement came from a “page stealing” technique that lets threads take ownership of pages without expensive cross-thread synchronization, developed in close collaboration with the Azure Cosmos DB team at Microsoft. The team says it will publish a technical report on these improvements soon.