Version 2

MLSys 2026 (May 18–22, Bellevue, WA) is wrapping up with four inference research threads now substantially documented. NVIDIA's BLASST (arXiv 2512.12087) won Best Paper for dynamic blocked sparse attention[3], and the community is already requesting its integration into FlashInfer[5]. Attention-FFN disaggregation has advanced rapidly from a conference mention to a concrete engineering push: StepFun AI published StepMesh, a dedicated communication library[7], vLLM posted an active RFC[8], and formal papers address both provisioning methodology[10] and hardware challenges[11]. Distribution-Aware Speculative Decoding, targeting long-tail RL rollout inefficiency, is confirmed at arXiv 2511.13841 with Together.ai claiming up to 50% acceleration of RL rollouts[20][19].

MLSys 2026 marks a moment when three inference techniques — dynamic sparse attention, disaggregated serving, and RL training efficiency via speculative decoding — are simultaneously crossing from research paper to production engineering artifact. The speed at which attention-FFN disaggregation moved from a single mention to competing libraries and formal RFCs within the same conference week illustrates how quickly the industrialization cycle now runs. Decisions being made in vLLM RFCs and FlashInfer issue trackers this week will define AI serving infrastructure for the next hardware generation.

MLSys 2026, held May 18–22 in Bellevue, Washington, brought together researchers and engineers from across the AI industry to address production inference and training systems challenges[1][2]. The conference's most prominent recognition went to NVIDIA's BLASST paper (arXiv 2512.12087), which introduces Dynamic Blocked Attention Sparsity via Softmax Thresholding[3]. BLASST dynamically identifies and skips low-salience attention blocks at runtime using a softmax threshold gate, directly attacking the core inefficiency of dense attention at long context lengths. Its Best Paper award reflects a broader productionization wave that SemiAnalysis documented in a pre-conference thread: sparse attention has migrated from academic benchmarks into live deployments, with DeepSeek's Sparse Attention and NousResearch's Lighthouse Attention cited as examples[4]. The inference community has already opened a feature request to integrate BLASST into FlashInfer[5], the widely used GPU kernel library, which would extend its reach beyond NVIDIA's own toolchain.

Prefill-decode (PD) disaggregation — routing the compute-bound prefill phase and the memory-bandwidth-bound decode phase to dedicated hardware pools — is now widely accepted as an industry best practice[6]. Attention-FFN disaggregation has emerged as the next architectural axis during this conference, with a speed that illustrates how quickly the industrialization cycle now runs. StepFun AI published StepMesh, a communication library purpose-built for attention-FFN disaggregation in MoE inference[7]. The vLLM project posted an RFC proposing ATTN-FFN disaggregation for MoE models[8][9], and formal research papers have appeared addressing both analytical provisioning for attention-FFN disaggregated serving under stochastic workloads[10] and the hardware systems challenges the approach surfaces for modern MoE architectures[11]. @haoailab (Hao Zhang's DistServe group), credited with originating PD disaggregation, is presenting a further disaggregation technique at the conference[6]. Alongside the research, production deployment activity is broad: LMCache Lab released an async PDBackend[12], Theta EdgeCloud published PD disaggregation test results[13][14], and financial commentators picked up the architecture as evidence for extended GPU hardware ROI — arguing that older prefill-heavy GPUs can be repurposed into disaggregated decode pools[15][16]. Practitioner consensus has hardened around co-located prefill and decode at scale being actively wasteful[17], though a visible tension remains: vLLM's own documentation still labels disaggregated prefilling as experimental[18].

Distribution-Aware Speculative Decoding (DASD) addresses a structural inefficiency in reinforcement learning post-training: the long-tail distribution of rollout lengths creates GPU bubbles that degrade training throughput. Together.ai quantifies the benefit at up to 50% acceleration of RL rollouts[19]. The paper (arXiv 2511.13841), from the WukLab group and presented at MLSys 2026, targets the mismatch between speculative decoding draft models — which assume fixed output length — and the highly variable rollout lengths characteristic of RL training[20][21]. A broader ecosystem of related work is emerging in parallel: MIT published on adaptive drafter-based RL training efficiency[22], and a FAISys 2025 paper tackled dynamic and online speculative decoding for RL[23]. SnorkelAI presented on reinforcement learning from verifiable rewards (RLVR) in low-data, low-compute settings at the conference[24].

For mixture-of-experts (MoE) models — now dominant at the frontier — expert load balancing at serving time remains a significant underexplored challenge. SemiAnalysis noted that this problem surfaces primarily in closed production systems and receives disproportionately little open-source attention[25]. A cross-layer load balancing paper for distributed MoE inference appeared at ICML 2026[26], and load-balancing losses applied during MoE training complicate runtime scheduling decisions[27], but no production-grade open-source solution has been publicly announced. Industry presence at MLSys 2026 was broad: vLLM, LMSYS, Inferact (co-hosting a luncheon with a16z), and Delta Institute were among the organizations attending[28][29][30][31].