Transformer Attention: A Decade of Innovation Recognized by SemiAnalysis

SemiAnalysis published a community recognition thread on June 29, 2026, tracing transformer attention from the 2017 Multi-Head Attention paper through FlashAttention, PagedAttention/vLLM, and the current wave of linear and sparse attention architectures. [10] The thread identifies four major contributions: the foundational 2017 Transformer paper by Vaswani, Shazeer, Jones, Gomez and colleagues [1]; FlashAttention by Tri Dao, which cut GPU memory requirements and enabled long-context training [2]; vLLM's PagedAttention for inference serving [5]; and a recent wave of linear and sparse attention research driven by agentic AI's long-context demands, led by Gated Delta Networks (GDNs) and DeepSeek's sparse attention work. [6] GDNs, developed by Songlin Yang, have been adopted by Qwen 3.5, with Kimi building further improvements on top. [6]

Attention mechanisms are the architectural core of every major language model, and this thread maps which open-source contributions actually made large-scale training and inference practical for the broader community. The field has not converged on a single attention design: linear attention (GDNs) and sparse attention (DeepSeek DSA) are now competing approaches for the long-context workload that standard quadratic MHA cannot handle efficiently.

The 2017 paper "Attention Is All You Need" by Ashish Vaswani, Noam Shazeer, Llion Jones, Aidan Gomez, and colleagues introduced Multi-Head Attention (MHA) to NLP and produced immediate, large improvements in perplexity scores over prior sequence models. [1] MHA became the dominant building block for language models, but its quadratic cost in sequence length created a ceiling on context length and on memory efficiency during training runs.

FlashAttention, developed by Tri Dao, addressed the memory bottleneck by restructuring attention computation to minimize GPU memory reads and writes rather than reducing FLOPs, cutting memory requirements for both forward and backward passes and enabling training on substantially longer contexts. [2] Three successive versions of FlashAttention have since been released, each targeting newer GPU architectures. [2][3] FlashAttention received the inaugural Stanford open source software award, a recognition of its practical impact across the community. [4] Around the same period, the vLLM inference engine introduced PagedAttention, a memory management approach that improved GPU utilization for serving. vLLM grew into one of the most widely used open-source inference engines, sustained by maintainers from Inferact and Red Hat. [5]

More recently, agentic AI applications requiring very long contexts drove a wave of research into architectural alternatives to quadratic MHA. Gated Delta Networks (GDNs), developed by Songlin Yang, combined the delta rule with gating mechanisms to allow more selective memory updates than earlier linear attention designs, and became the leading linear attention approach. [6][7][8] GDNs were adopted by Qwen 3.5, and Kimi built further refinements on top. [6] A follow-up paper, Gated DeltaNet-2, decouples the erase and write operations to further improve memory control. [9] In parallel, DeepSeek led open research into sparse attention with Native Sparse Attention and DeepSeek Sparse Attention (DSA), work that directly influenced related efforts at MiniMax and ZhipuAI. Cohere popularized SWA-GQA hybrid attention, which Xiaomi later analyzed with detailed ablation studies. [6]

SemiAnalysis framed this full arc as collective open-source progress, explicitly crediting individual researchers and teams rather than any single institution. [10] The thread positions attention research as an ongoing field where inference infrastructure (FlashAttention, vLLM) and architectural innovation (GDNs, sparse attention) have both been essential—and where no single design has yet displaced the others across all use cases.