從 GPT-2（2019）到 KimiK3（2026）的架構演進，展現了高達 22,580 倍的規模擴張。基礎的 GPT-2 僅具備解碼器架構，包含約 1.24 億個參數（配置為 50k 詞彙量、12 個區塊、12 個注意力頭與 768 維嵌入空間）；而 KimiK3 的規模已躍升至 2.8 兆個參數。此龐大的擴展凸顯了標準 O(N²) Softmax 注意力機制中固有的記憶體頻寬瓶頸；在該機制中，為先前序列儲存鍵（Key）與值（Value）向量，會導致記憶體讀寫需求呈現無法負荷的線性增長。

為克服二次方複雜度與加法性干擾，模型架構歷經 Linear Attention 與 DeltaNet 的演進。Linear Attention 透過分塊運算（通常將區塊大小設定為 C=64 或 C=128，以最佳化張量核心的使用率），將 Query 與 Key 分別映射並折疊至固定的 D×D 狀態空間中，從而提升效率。然而，有限的記憶體容量必須搭配替換策略。Gated DeltaNet 以及隨後的 Kimi Linear 導入了局部衰減參數，以選擇性地清除舊有記憶。藉由實作細粒度的通道級別記憶衰減控制，這些系統在維持高保真度的同時，大幅降低了計算負擔，並使解碼吞吐量提升高達 6 倍。

KimiK3 將上述技術進展整合為高度專業化的混合架構，其包含 23 個四層巨集週期（Macrocycles）。每個週期交替使用 3 層提供固定狀態遞迴的 Kimi Delta Attention（KDA），以及 1 層用於完整 Softmax 檢索的 Multi-head Latent Attention（MLA）。此外，該模型採用稀疏混合專家（MoE）系統，共計 898 個專家網路（包含 2 個共享專家，每個 token 路由至 16 個專家），並結合 SiTU 激勵函數，使活躍的浮點運算次數（FLOPs）減少近半。最後，模型每 12 層部署一次區塊級別的注意力殘差（AttnRes）——總計 8 個區塊——此設計僅增加約 2% 的推論延遲，卻藉由允許選擇性存取早期的深度表徵，帶來了 1.25 倍的運算優勢。

The architectural evolution from GPT-2 (2019) to KimiK3 (2026) represents a scaling factor of exactly 22,580. While the foundational GPT-2 decoder-only model contained roughly 124 million parameters (configured with a 50k vocabulary size, 12 blocks, 12 heads, and a 768 embedding dimension), KimiK3 has expanded to a massive 2.8 trillion parameters. This vast expansion highlights the memory-bandwidth bottleneck inherent in standard O(N²) softmax attention, where storing Key and Value (KV) vectors for prior sequence tokens creates unsustainable linear growth in memory reading and writing demands.

To overcome quadratic complexity and additive interference, the architecture evolved through Linear Attention and DeltaNet. Linear Attention achieves efficiency by chunking operations (typically at sizes of C=64 or C=128 to optimize tensor-core utilization), mapping queries and keys separately to fold them into a fixed D×D state. However, finite memory capacity necessitates an eviction policy. Gated DeltaNet and subsequently Kimi Linear introduced a localized decay parameter to selectively clear old memory. By implementing fine-grained, per-channel memory decay controls, these systems maintain high fidelity while substantially reducing computational overhead and achieving up to a 6x increase in decoding throughput.

KimiK3 synthesizes these advancements into a highly specialized hybrid structure comprising 23 four-layer macrocycles. Each macrocycle interleaves three Kimi Delta Attention (KDA) layers for constant-state recurrence with one Multi-head Latent Attention (MLA) layer for full softmax retrieval. Furthermore, the model utilizes a sparse Mixture-of-Experts (MoE) system with 898 total experts (2 shared, 16 active per token) and SiTU activations, nearly halving active FLOPs. Finally, it implements Blockwise Attention Residuals (AttnRes) every 12 layers—totaling 8 blocks—which adds a mere 2% inference latency while delivering a 1.25x compute advantage by allowing selective access to earlier depth-wise representations.