Kimi Linear which is a hybrid attention model designed to replace standard full attention while fixing the extreme memory and compute bottlenecks that happen during long context tasks and reinforcement learning. They built a custom linear attention module called Kimi Delta Attention and interleaved it with standard Multi-Head Latent Attention layers at a three to one ratio. It turns out that their custom layer improves on earlier gated linear models by using a channel-wise forget gate instead of a coarse head-wise one. The fine-grained gating gives the model much better control over its recurrent memory. They also came up with a chunkwise parallel algorithm that makes it incredibly hardware efficient compared to older matrix formulations.
A 48 billion parameter mixture of experts model with 3 billion active parameters and trained on 1.4 trillion tokens consistently beat a pure full attention baseline and another hybrid baseline across general knowledge, math, and coding benchmarks. Because of the three to one hybrid structure it cuts key-value cache memory usage by up to 75%, and scaling up to a one million token context window Kimi Linear hits a decoding throughput that is six times faster than standard full attention. It also showed superior scaling and convergence during reinforcement learning phases. Extending the training to 5.7 trillion tokens scored a 94.8 on the RULER benchmark at a one million context length. Getting this level of performance while drastically slashing memory requirements makes hybrid linear architectures a strong candidate for the next generation of reasoning models.