Contents
Kimi K3: The Open-Source 2.8T MoE Model That Rethinks Attention
On July 27, 2026, Moonshot AI released Kimi K3 — a 2.8-trillion-parameter Mixture-of-Experts model with 104B activated parameters per token, a 1M-token context window, and a novel attention mechanism called Kimi Delta Attention (KDA). With ~400 authors listed on the paper (arXiv:2607.24653), K3 is fully open-source under a permissive license. This article breaks down what KDA is, how K3 is built, and why it matters.
1. The Problem: Attention at Scale
Standard softmax attention scales quadratically with sequence length. At 1M tokens, the KV cache for a 93-layer transformer consumes hundreds of gigabytes — making long-context inference prohibitively expensive. Linear attention mechanisms (RWKV, Mamba, DeltaNet) solve the memory problem but historically underperform softmax attention on recall-intensive tasks.
K3’s central insight: linear attention doesn’t have to trade off quality. KDA is the first linear attention variant that matches or outperforms full softmax attention under fair comparisons, while slashing the KV cache by 75% and achieving 6× faster decoding at 1M context.
2. Kimi Delta Attention (KDA)
KDA extends Gated DeltaNet (GDN) with a critical change: per-channel decay instead of scalar decay. The recurrence is:
S_t = (I - β_t k_t k_t^T) · Diag(α_t) · S_{t-1} + β_t k_t v_t^TWhere Diag(α_t) is a diagonal matrix enabling each hidden dimension to decay independently. GDN uses a single scalar α_t — one forget gate for the entire state. KDA’s per-channel gating lets the model decide which dimensions to retain and which to forget, at a granular level. This is the key to KDA’s recall parity with softmax attention.
Attention Residuals (AttnRes) further improve information flow across model depth by allowing layers to selectively retrieve past representations — a skip-connection for attention states.
Original KDA paper
KDA was first introduced in Kimi Linear (arXiv:2510.26692). The K3 paper applies it at production scale — 93 layers, 896 experts, training on trillions of tokens.3. Hybrid Architecture: KDA + MLA
K3 uses a hybrid attention stack: 93 total layers, with 69 KDA layers and 24 Gated MLA layers (Multi-head Latent Attention) + 1 initial dense layer. The ratio is roughly 3 KDA layers for every 1 MLA layer — 75% of the model uses linear attention.
This hybrid gives K3 the best of both worlds: KDA handles long-range retrieval efficiently, while MLA preserves the precision of full attention where it matters most (typically earlier and later layers). The full architecture:
| Parameter | Value |
|---|---|
| Total parameters | 2.8T |
| Activated per token | 104B |
| Layers | 93 (69 KDA + 24 Gated MLA + 1 dense) |
| Attention heads | 96 |
| Hidden dim | 7,168 |
| Routed experts | 896 |
| Shared experts | 2 |
| Active experts per token | 16 |
| MoE hidden dim per expert | 3,072 |
| Context window | 1,048,576 (1M) |
| Vocabulary | 160K tokens |
| Vision encoder | MoonViT-V2 (401M params) |
| Quantization | MXFP4 weights / MXFP8 activations |
| Activation | SiTU-GLU |
4. Stable LatentMoE
K3 introduces Stable LatentMoE, a refinement of the MoE routing mechanism with two key innovations:
- Quantile Balancing: Routes tokens based on quantile scores instead of raw router logits, removing sensitive balancing hyperparameters that plague traditional MoE training.
- Per-Head Muon optimizer: Applies the Muon optimizer independently per attention head, stabilizing large-scale distributed training.
Combined with SiTU activation (an improvement over SwiGLU for MoE architectures) and Gated MLA integration, Stable LatentMoE ensures all 896 experts are utilized efficiently throughout training without collapse.
5. Scaling Efficiency
K3 achieves approximately 2.5× scaling improvement over Kimi K2. This comes from the combination of four innovations:
- KDA: 6× faster decoding, 75% less KV cache
- AttnRes: Better gradient flow across depth, enabling more effective training
- Stable LatentMoE: Better expert utilization and training stability
- Improved data recipes: Higher-quality training mixtures and curriculum strategies
At 104B activated parameters, K3 offers GPT-5-class intelligence at a fraction of the inference cost. The MXFP4 weight quantization (with quantization-aware training) keeps memory footprint manageable even for the full 2.8T model.
6. Benchmark Results
K3 was evaluated against GPT-5.6 Sol, Fable 5, and Opus 4.8 across coding, reasoning, agentic, and vision benchmarks. Wins are highlighted in green, losses in red.
| Benchmark | Kimi K3 | Fable 5 | GPT-5.6 Sol | Opus 4.8 |
|---|---|---|---|---|
| Coding | ||||
| DeepSWE | 67.5 | 70.0 | 73.0 | 59.0 |
| SWE-Marathon | 42.0 | 35.0 | 39.0 | 40.0 |
| FrontierSWE | 81.2 | 86.6 | 71.3 | 66.7 |
| Terminal-Bench 2.1 | 88.3 | 88.0 | 88.8 | 84.6 |
| ProgramBench | 77.8 | 76.8 | 77.6 | 71.9 |
| KCB 2.0 | 72.9 | 76.9 | 64.8 | 71.7 |
| Agentic | ||||
| BrowseComp | 91.2 | 88.0 | 90.4 | 84.3 |
| DeepSearchQA (F1) | 95.0 | 94.2 | — | 93.1 |
| MCPMark-Verified | 94.5 | 87.4 | 92.9 | 76.4 |
| AutomationBench | 30.8 | 29.1 | 29.7 | 27.2 |
| Vision | ||||
| WorldVQA | 51.0 | 56.7 | 41.8 | 39.1 |
| Video-MME (w/ sub) | 90.0 | — | 89.5 | 86.0 |
| OmniDocBench | 91.1 | 89.8 | 85.8 | 87.9 |
| Reasoning | ||||
| GPQA Diamond | 93.5 | 92.6 | 94.1 | 91.0 |
| OSWorld 2.0 | 58.3 | 66.1 | 62.6 | 55.7 |
K3 leads on SWE-Marathon, BrowseComp, DeepSearchQA, MCPMark-Verified, ProgramBench, Video-MME, and OmniDocBench. It trails Fable 5 on DeepSWE, FrontierSWE, KCB 2.0, WorldVQA, and OSWorld 2.0 — primarily on agentic benchmarks requiring complex GUI interaction. Notably, K3 beats GPT-5.6 Sol on 12 of 14 comparable benchmarks.
BrowseComp note
K3’s 91.2 on BrowseComp uses context compaction at ~300K tokens. With full 1M native context, K3 scores 90.4 — still competitive.7. Post-Training & RL
K3 undergoes reinforcement learning across three domains: general reasoning, agentic tasks, and coding. The model supports multiple reasoning-effort levels — low, high, and max (default) — letting users trade latency for quality. K3 always has thinking enabled and returns reasoning_content in its responses.
An important deployment detail: K3’s thinking history mode requires the full assistant message (including reasoning_content) to be passed back on subsequent turns. Agent harnesses that strip this field will see degraded quality.
8. Open-Source Impact
K3 is fully open-sourced under the Kimi K3 License. The repository (github.com/MoonshotAI/Kimi-K3) includes:
- Full model weights (2.8T parameters)
- Training and inference code
- Integration with vLLM, SGLang, and TokenSpeed inference engines
- Vision encoder (MoonViT-V2)
This is one of the largest fully open-source models ever released — at a scale typically reserved for closed APIs. Moonshot AI’s pricing reflects this accessibility: cache-hit input at $0.30/MTok, cache-miss at $3.00/MTok, and output at $15.00/MTok. The model is accessible via platform.kimi.ai (model: kimi-k3) with an OpenAI/Anthropic-compatible API.
For developers, the recommended agent framework is Kimi Code CLI.
9. Case Studies: Beyond Benchmarks
The K3 paper documents several remarkable autonomous workflows that demonstrate the model’s agentic capabilities:
- Chip design: K3 designed its own ASIC in 48 hours — a 1.46M standard-cell design with 0.277 MB SRAM, targeting 100 MHz.
- MiniTriton: Built a GPU compiler from scratch, demonstrating low-level systems programming competence.
- Video editing: Edited a teaser trailer from 56 raw clips with motion-matched cuts and beat-synced transitions.
- Research automation: Reproduced complex astrophysics research in 2 hours — work that normally takes 1–2 weeks.
- GPU kernel optimization: Auto-profiled and rewrote its own attention kernels for better performance.
These case studies signal a shift: K3 isn’t just a chatbot — it’s an autonomous research assistant capable of end-to-end engineering workflows.
10. Limitations
The paper is transparent about K3’s weaknesses:
- Sensitivity to thinking history: If an agent harness doesn’t pass the full assistant message (including
reasoning_content), quality degrades significantly. This is a deployment gotcha, not a model flaw, but it matters. - Excessive proactiveness: K3 sometimes makes autonomous decisions when user intent is ambiguous. This is great for power users but can surprise less technical users.
- UX gap: The paper acknowledges a gap vs Fable 5 and GPT-5.6 Sol in overall user experience — likely reflecting product polish rather than raw capability.
11. Conclusion
Kimi K3 is a landmark release on multiple fronts. It proves that linear attention (KDA) can match full attention at scale — a result many thought impossible. It demonstrates that open-source models can compete with closed frontier systems across coding, reasoning, and agentic benchmarks. And its 2.8T parameters, fully open-sourced, set a new bar for transparency in AI research.
The combination of KDA’s memory efficiency, Stable LatentMoE’s training stability, and aggressive quantization (MXFP4 weights) makes K3 both capable and practical to deploy. While it trails Fable 5 on a handful of agentic benchmarks, K3’s lead on MCPMark-Verified (+7.1), BrowseComp (+3.2), and Video-MME (+0.5 over GPT-5.6 Sol) shows that efficient attention architectures are not a compromise — they’re an advantage.
For researchers, KDA opens a new direction in efficient attention. For developers, K3 offers a powerful, cost-effective model that works with existing tooling. For the field, it’s a reminder that the gap between open and closed AI is narrowing faster than most expected.
Paper: arXiv:2607.24653 · Code: github.com/MoonshotAI/Kimi-K3 · API: platform.kimi.ai (model: kimi-k3)
