从 GPT-2（2019）到 KimiK3（2026）的架构演进，展现了高达 22,580 倍的规模扩张。基础的 GPT-2 仅具备解码器架构，包含约 1.24 亿个参数（配置为 50k 词汇量、12 个区块、12 个注意力头与 768 维嵌入空间）；而 KimiK3 的规模已跃升至 2.8 兆个参数。此庞大的扩展凸显了标准 O(N²) Softmax 注意力机制中固有的记忆体频宽瓶颈；在该机制中，为先前序列储存键（Key）与值（Value）向量，会导致记忆体读写需求呈现无法负荷的线性增长。

为克服二次方复杂度与加法性干扰，模型架构历经 Linear Attention 与 DeltaNet 的演进。Linear Attention 透过分块运算（通常将区块大小设定为 C=64 或 C=128，以最佳化张量核心的使用率），将 Query 与 Key 分别映射并折叠至固定的 D×D 状态空间中，从而提升效率。然而，有限的记忆体容量必须搭配替换策略。Gated DeltaNet 以及随后的 Kimi Linear 导入了局部衰减参数，以选择性地清除旧有记忆。借由实作细粒度的通道级别记忆衰减控制，这些系统在维持高保真度的同时，大幅降低了计算负担，并使解码吞吐量提升高达 6 倍。

KimiK3 将上述技术进展整合为高度专业化的混合架构，其包含 23 个四层巨集周期（Macrocycles）。每个周期交替使用 3 层提供固定状态递回的 Kimi Delta Attention（KDA），以及 1 层用于完整 Softmax 检索的 Multi-head Latent Attention（MLA）。此外，该模型采用稀疏混合专家（MoE）系统，共计 898 个专家网路（包含 2 个共享专家，每个 token 路由至 16 个专家），并结合 SiTU 激励函数，使活跃的浮点运算次数（FLOPs）减少近半。最后，模型每 12 层部署一次区块级别的注意力残差（AttnRes）——总计 8 个区块——此设计仅增加约 2% 的推论延迟，却借由允许选择性存取早期的深度表征，带来了 1.25 倍的运算优势。

The architectural evolution from GPT-2 (2019) to KimiK3 (2026) represents a scaling factor of exactly 22,580. While the foundational GPT-2 decoder-only model contained roughly 124 million parameters (configured with a 50k vocabulary size, 12 blocks, 12 heads, and a 768 embedding dimension), KimiK3 has expanded to a massive 2.8 trillion parameters. This vast expansion highlights the memory-bandwidth bottleneck inherent in standard O(N²) softmax attention, where storing Key and Value (KV) vectors for prior sequence tokens creates unsustainable linear growth in memory reading and writing demands.

To overcome quadratic complexity and additive interference, the architecture evolved through Linear Attention and DeltaNet. Linear Attention achieves efficiency by chunking operations (typically at sizes of C=64 or C=128 to optimize tensor-core utilization), mapping queries and keys separately to fold them into a fixed D×D state. However, finite memory capacity necessitates an eviction policy. Gated DeltaNet and subsequently Kimi Linear introduced a localized decay parameter to selectively clear old memory. By implementing fine-grained, per-channel memory decay controls, these systems maintain high fidelity while substantially reducing computational overhead and achieving up to a 6x increase in decoding throughput.

KimiK3 synthesizes these advancements into a highly specialized hybrid structure comprising 23 four-layer macrocycles. Each macrocycle interleaves three Kimi Delta Attention (KDA) layers for constant-state recurrence with one Multi-head Latent Attention (MLA) layer for full softmax retrieval. Furthermore, the model utilizes a sparse Mixture-of-Experts (MoE) system with 898 total experts (2 shared, 16 active per token) and SiTU activations, nearly halving active FLOPs. Finally, it implements Blockwise Attention Residuals (AttnRes) every 12 layers—totaling 8 blocks—which adds a mere 2% inference latency while delivering a 1.25x compute advantage by allowing selective access to earlier depth-wise representations.