Kog's Laneformer 2B hits 3,000 tokens/s by making latency the whole point

Most large language model (LLM) research chases benchmark scores first, treating inference speed as a downstream serving problem, something to be fixed later with quantization, sharding, batching, or caching. Kog, a Paris-based AI infrastructure startup, flipped that script entirely. Speed was the primary design goal from day one.

The result is Laneformer 2B, a 2.3 billion-parameter Transformer built for single-request, low-latency decoding. Released under an open Apache License 2.0, the model and its architectural innovations show how hardware constraints can reshape neural network design, particularly for startups with tight resources.

Architecture co-designed with the engine

At low batch sizes, typical for agents or interactive apps, decode speed isn't just about floating-point operations. A big chunk of time goes to moving model weights, synchronizing kernels, and shuffling data between GPUs at every layer. Traditional Tensor Parallelism (TP) forces devices to exchange results layer by layer, a heavy drag during one-sample decoding.

To get around that, Kog devised Delayed Tensor Parallelism (DTP), a trick that delays inter-GPU synchronization by two steps, effectively sweeping communication latency under the rug. That required a custom eight-lane architecture, where each lane handles a slice of the hidden dimension and attention heads. This lane structure lets DTP work without hurting model quality. It's the most significant departure from the standard decoder-only Transformer.

The model uses causal Grouped-Query Attention (GQA) with 32 query heads and 16 key/value heads, evenly split across the eight lanes. To cut KV cache streaming lag, 10 of the 15 layers rely on Sliding Window Attention (SWA). The context length is capped at 4,096 tokens, a deliberate choice that balances speed and usefulness for code generation.

Training on 6 trillion tokens with open datasets

Laneformer 2B was trained from scratch in two phases on a cluster of 192 NVIDIA H100 GPUs across 24 nodes, courtesy of Scaleway and ADASTRA. The training stack used TorchTitan for distributed FSDP, achieving roughly 17,000 tokens per second per GPU over about 21 days.

The data recipe drew heavily from NVIDIA's Nemotron pre-training datasets, but with a twist: unlike typical multi-phase training, Kog deliberately shifted the data mixture sharply between phases. Phase 1 consumed roughly 4 trillion tokens of broad generalist data. Phase 2 added about 2 trillion tokens heavily weighted toward code and reasoning. Post-training used around 210 million tokens for instruction and identity fine-tuning.

"This was an explicit trade-off," the Kog team wrote in the blog post. "A strong data-mixture shift between phases is not standard practice, and we expected it to hurt some general capabilities." DataComp-LM Core centered accuracy tracking confirmed a non-negligible hit to general skills, but the team decided it was a fair price for a strong coding-focused model.

Benchmark results against similar-size models

Using the official evalplus runner with greedy decoding (temperature 0, do_sample=False), Laneformer 2B scored 45.1% on HumanEval+ and 51.6% on MBPP+. Those numbers put it at the competitive edge of the 2B-parameter class. The Phase 2 data shift alone boosted coding performance by more than 10 points on HumanEval+.

The team also noted that pass@N scores climb consistently for N = 2, 4, 8, and 16, making stochastic decoding a practical way to trade a bit of latency for higher accuracy.

Inference speed claims

In the public Kog Inference Engine (KIE) preview, Laneformer 2B hits 3,000 output tokens per second per request on 8x AMD MI300X and 2,100 tokens/s on 8x NVIDIA H200, using FP16, batch size 1, and no speculative decoding. The team claims this is the fastest publicly demonstrated single-request decoding result for a 2B-class model on standard datacenter GPUs.

However, the team acknowledges that the released Hugging Face checkpoint, while usable as a standard model, won't reach those speeds without the DTP-aware execution path in KIE. Standard Transformer implementations won't benefit from the latency optimizations.

Limitations and future work

Laneformer 2B isn't a general-purpose frontier model. Its 4,096-token context window and SWA choices reflect the latency target. Long-context extension is already in the works. The Phase 2 specialization also means general capabilities, on tasks like factual recall or broad QA, lag behind. The team plans a stronger post-training pipeline for future versions.

Why open-source?

Kog released the weights, custom Hugging Face implementation, configuration, evaluation results, and documentation under the Apache License 2.0. The tokenizer, based on Llama 2, is distributed under the Llama 2 Community License. The move positions Laneformer 2B as both a usable checkpoint and a research artifact for the open-source community.

Industry implications

Laneformer 2B shows that for startups without the resources to train large frontier models, a focused strategy, optimizing for a single metric (latency) and a specific domain (code generation), can produce competitive results. By open-sourcing both the model and the architectural recipe, Kog invites the community to build on its ideas and explore latency-first design further.

The model's strong performance relative to its size and training budget suggests that architecture co-design with hardware constraints could become a more common methodology, especially as inference costs and latency requirements become critical for real-time applications like AI agents.