Moonshot AI unveils Kimi K3, a 2.8T open MoE model with 1M context
Moonshot AI has just pushed open frontier-model boundaries again with Kimi K3, the first open 3-trillion-parameter Mixture-of-Experts (MoE) model to reach 2.8 trillion active parameters, native vision support, and a native context window of one million tokens. The team positions K3 as a leap toward long-horizon coding, complex knowledge work, and multi-step reasoning, while acknowledging that its overall performance still trails today’s strongest proprietary systems.
Under the hood: Kimi Delta Attention and Attention Residuals
K3’s architecture centers on two new mechanisms: Kimi Delta Attention (KDA), a hybrid linear attention layer that Moonshot claims accelerates decoding by up to 6.3× in million-token contexts, and Attention Residuals (AttnRes), which selectively retrieves representations across model depth instead of accumulating them uniformly. Together with Stable LatentMoE—activating just 16 of 896 experts via Quantile Balancing—the stack is designed to improve both throughput and training efficiency, delivering roughly 2.5× better overall scaling than its predecessor K2. On the deployment side, Moonshot baked quantization-aware training from the start, using MXFP4 weights and MXFP8 activations for broad hardware compatibility and recommending supernode clusters with 64 or more accelerators.
Benchmarks and caveats
Across Moonshot’s evaluation suite, K3 consistently outperforms other tested open models, but direct comparisons with proprietary systems reveal gaps. In published scores, K3 leads on Program Bench, SWE Marathon, BrowseComp, Automation Bench, and OmniDocBench, trails Fable 5 on FrontierSWE and HLE-Full, and sits behind GPT 5.6 Sol on DeepSWE. Moonshot notes that some benchmarks apply fallback rules or context compaction, which can shift rankings. The team emphasizes that all K3 results use maximum reasoning effort and caution that these scores should be read with the usual benchmark caveats in mind.
Why it matters
Kimi K3’s arrival signals that open models can now scale into the multi-trillion-parameter regime while supporting million-token contexts and native vision, democratizing capabilities once confined to closed systems. For developers and researchers, the open release lowers the barrier to experimenting with long-context reasoning and large-scale MoE architectures. For the industry, it underscores a maturing open ecosystem that can iterate quickly on architectural innovations like Kimi Delta Attention and Attention Residuals, narrowing the performance gap with proprietary leaders while remaining transparent and reproducible.
Source: MarkTechPost. AI-assisted editorial synthesis — TechnoExpress.