Ma Jiaqi taught MiniMax engineers a hard lesson about forgotten tokens

In recent weeks, developers using MiniMax's M2 series of large language models stumbled upon a curious and reproducible bug. The model could answer factual questions about Chinese singer Ma Jiaqi, naming his group or debut year, but it could not generate the token sequence "嘉祺" (Jiaqi) itself. The anomaly ignited discussions on Chinese platforms like Xiaohongshu and Zhihu, where community developers ran their own rigorous analyses, including tokenizer comparisons and sampling parameter tests.

Initial hypothesis: tokenizer mismatch ruled out

The first suspect was tokenizer misalignment between pre-training and post-training. Using the post-training tokenizer, "马嘉祺" encodes as token IDs [4143, 190467], where "嘉祺" became a single token (ID 190467). The obvious question: was this token properly trained during pre-training, or was it a new token added only later?

MiniMax engineers examined the pre-training model's vocabulary embedding layer. They found that token 190467's nearest neighbors in embedding space, including "亚轩", "千玺", "祺", "耀文", and "王一博", were all plausible person-name tokens. Statistical analysis confirmed the token was well-trained and semantically coherent in the base model, ruling out the tokenizer mismatch hypothesis.

Identifying the true cause: post-training data distribution

Searching the post-training dataset for occurrences of token 190467 turned up fewer than five samples containing "嘉祺". That scarcity was the smoking gun: the token was effectively unlearned during supervised fine-tuning (SFT) due to insufficient representation.

But why did the model retain comprehension while losing generation ability? The answer lies in the asymmetric behavior of the two layers that directly interface with token-level mapping: the input embedding layer (vocab embedding) and the output projection layer (lm_head).

Vocab embedding: nearly unchanged

Comparing pre-training and SFT vocab embeddings showed almost zero difference. Gradient norms decay through the Transformer layers, and for extremely rare tokens, the embedding layer receives negligible gradient updates from the loss function. Only weight decay exerts a mild regularizing effect, leaving the input embedding essentially frozen.

lm_head: significant drift

In contrast, the lm_head, the linear layer that projects hidden states to token probabilities at each generation step, showed dramatic changes. The weight vector for "嘉祺" shifted substantially during SFT, as measured by L2 distance between pre-training and post-training vectors. This drift caused the token's generation probability to fall below the top-p=0.95 sampling threshold commonly used during inference, effectively silencing it.

Systematic analysis: which tokens drift most?

To understand whether this was an isolated case, MiniMax computed L2 differences for every token in the vocabulary across the entire SFT process, sorting them by magnitude of change. The categories most affected:

1. Special tokens (expected)
2. Japanese colloquialisms and web templates (~40% of highest-drift tokens)
3. LaTeX and web metadata tokens
4. Chinese SEO and spam text

The dominance of Japanese tokens was unexpected but immediately explained a longstanding bug in the M2.5 model: occasional language mixing in Japanese conversational contexts, outputs that would suddenly switch to Russian or Korean mid-sentence. This had been classified as a separate "small language mixing" issue without a known root cause.

The analysis unified both problems under one framework: insufficient coverage of Japanese tokens in post-training data caused their lm_head vectors to drift and mix with other languages' token representations in vector space. Low-frequency Chinese tokens like "嘉祺" that happened to be spatially adjacent to drifted Japanese tokens were also pushed out of the high-probability zone, the mechanism of sparse token forgetting.

Repair experiment: full-vocabulary coverage

MiniMax designed a minimal intervention: synthetic repetition data covering every token in the vocabulary, appended as a small fraction of the standard SFT dataset. The synthetic task was simple, each token appeared once as the target for a "copy this text" prompt, ensuring every token received at least one gradient update as a generation target during training.

Results

The quantitative validation: experiment group maintained lm_head cosine similarity to the pre-training base at mean 0.9992, with minimum 0.9711 and zero tokens below 0.95. Baseline mean cos_sim was 0.9837, with 9,805 tokens (4.9%) below 0.95 and 4,234 (2.1%) below 0.90. Japanese tokens were hit hardest: mean cos_sim 0.9502, with 29.7% below 0.95.

Broader implications

The structural mismatch between pre-training vocabulary and post-training data distribution is not unique to MiniMax. As models grow, tokenizers are trained on massive, diverse corpora that inevitably include rare or language-specific tokens. Post-training datasets, by contrast, are curated for instruction-following and safety, and their distribution naturally undersamples low-frequency tokens.

MiniMax argues that post-training data quality should be measured along two dimensions: semantic diversity (task and domain coverage) and token-level coverage (ensuring every token appears as a generation target at least once). Adding token coverage as a routine monitoring metric could prevent similar issues from appearing in production.

With the release of other models like Opus 4.7 also announcing tokenizer changes, the industry is clearly aware that tokenizer design and post-training data alignment remain areas with significant optimization potential.

Conclusion

The sparse token forgetting phenomenon has a clear mechanism: during SFT, lm_head vectors for low-frequency tokens drift without being corrected by gradient updates, reducing their generation probability below usable thresholds. Input embeddings remain stable, preserving comprehension. A simple full-vocabulary coverage strategy, adding synthetic repetition data, effectively anchors all token representations, resolving both the Ma Jiaqi case and the Japanese language mixing bug.