Version 2

SemiAnalysis published a recognition thread on June 29, 2026 tracing transformer attention from the 2017 Vaswani et al. paper through FlashAttention, vLLM's PagedAttention, and the current split between linear and sparse attention research. [12] The four credited contributions are: the foundational Multi-Head Attention paper [1]; FlashAttention by Tri Dao [2]; vLLM's PagedAttention [5]; and a current contest between Gated Delta Networks (linear attention, adopted by Qwen 3.5 and Kimi) and DeepSeek's sparse attention work (NSA and DSA, now integrated into NVIDIA's cuDNN). [6][11] GDN-2, a follow-up that decouples erase and write in linear attention, is an NVIDIA Research publication. [10]

Attention mechanisms are the architectural core of every major language model, and this thread maps which open-source contributions made large-scale training and inference practical for the broader community. The field has not converged: linear attention (GDNs) and sparse attention (DSA, now in NVIDIA cuDNN) are competing strategies for long-context workloads that quadratic MHA handles inefficiently at scale.

The 2017 paper "Attention Is All You Need" by Ashish Vaswani, Noam Shazeer, Llion Jones, Aidan Gomez, and colleagues introduced Multi-Head Attention (MHA) to NLP, producing immediate large improvements in perplexity scores over prior sequence models. [1] MHA became the dominant building block for language models, but its quadratic cost in sequence length created a ceiling on context length and memory efficiency during training runs.

FlashAttention, developed by Tri Dao, addressed the memory bottleneck by restructuring attention computation to minimize GPU memory reads and writes rather than reducing FLOPs, enabling training on substantially longer contexts. [2] FlashAttention 2 achieved approximately 800% throughput improvement over the original; versions 3 and 4 targeted Hopper and Blackwell GPU architectures. [3] FlashAttention received the inaugural Stanford open source software award. [4] In the same period, the vLLM inference engine introduced PagedAttention, a memory management approach that improved GPU utilization for serving, and became one of the most widely used open-source inference engines, maintained by teams at Inferact and Red Hat. [5]

Agentic AI applications requiring very long contexts drove research into architectural alternatives to quadratic MHA. Gated Delta Networks (GDNs), developed by Songlin Yang, combined the delta rule with gating mechanisms for more selective memory updates than earlier linear attention designs, and became the leading linear attention approach. [6][7][8] GDNs were adopted by Qwen 3.5, with Kimi building further refinements. A follow-up, Gated DeltaNet-2, published through NVIDIA Research, decouples erase and write operations to give finer memory control. [9][10] In parallel, DeepSeek led open research into sparse attention with Native Sparse Attention and DSA, influencing MiniMax and ZhipuAI. NVIDIA's cuDNN now includes official documentation for DSA, providing production infrastructure support for the sparse attention path. [11] Cohere popularized SWA-GQA hybrid attention, which Xiaomi later refined with detailed ablation studies. [6]

SemiAnalysis framed this arc as collective open-source progress, crediting individual researchers and teams rather than single institutions. [12] NVIDIA has emerged as an actor on both fronts simultaneously — providing cuDNN infrastructure for DSA while its researchers co-produced the GDN-2 linear attention paper — without publicly backing either approach over the other.