Version 3

MLSys 2026 (May 18–22, Bellevue, WA) concluded with four inference research threads extending rapidly from conference papers into engineering artifacts. NVIDIA's BLASST (arXiv 2512.12087) won Best Paper for dynamic blocked sparse attention[3], and the FlashInfer community is tracking integration including sparse-MLA kernels for new SM120 hardware[6]. Attention-FFN disaggregation has produced competing implementations: StepFun AI's StepMesh is openly hosted on GitHub[8] and documented in the Step-3 system paper[9], while vLLM pursues an RFC[10]. Two independent groups — Together.ai/WukLab with DASD (arXiv 2511.13841)[22] at MLSys and MIT HAN Lab with Adaptive Drafter (arXiv 2511.16665, ASPLOS'26)[23][24] — have simultaneously targeted long-tail RL rollout inefficiency, with secondary sources attributing a 2x training speed improvement to the MIT approach[25].

MLSys 2026 marks a moment when sparse attention, disaggregated serving, and RL training efficiency via speculative decoding are all crossing from research papers into production engineering artifacts simultaneously. The convergence of two independent academic groups — Together.ai/WukLab and MIT HAN Lab — on the same long-tail RL rollout inefficiency signals that this problem is recognized as a high-value systems target worth parallel investment. The post-conference activity in vLLM RFCs and FlashInfer issue trackers will define AI serving infrastructure for the next hardware generation.

MLSys 2026, held May 18–22 in Bellevue, Washington, brought together researchers and engineers from across the AI industry to address production inference and training systems challenges[1][2]. The conference's most prominent recognition went to NVIDIA's BLASST paper (arXiv 2512.12087), which introduces Dynamic Blocked Attention Sparsity via Softmax Thresholding[3]. BLASST dynamically identifies and skips low-salience attention blocks at runtime using a softmax threshold gate, attacking the core inefficiency of dense attention at long context lengths. Its Best Paper award reflects a broader productionization wave documented in a pre-conference SemiAnalysis thread: sparse attention has migrated from academic benchmarks into live deployments, with DeepSeek's Sparse Attention and NousResearch's Lighthouse Attention cited as examples[4]. The inference community has opened a feature request to integrate BLASST into FlashInfer[5]; separately, FlashInfer's issue tracker shows active work on sparse-MLA paged attention for the new SM120 GPU architecture[6], indicating that sparse attention kernel development is continuing across multiple hardware targets in the weeks following the conference.

Prefill-decode (PD) disaggregation — routing the compute-bound prefill phase and the memory-bandwidth-bound decode phase to dedicated hardware pools — is now widely accepted as an industry best practice[7]. Attention-FFN disaggregation has emerged as the next architectural axis, with a speed of industrialization that illustrates how quickly the ML systems cycle now runs. StepFun AI's StepMesh communication library for attention-FFN disaggregated MoE inference is openly published on GitHub[8], and the accompanying Step-3 paper (arXiv 2507.19427) documents model-system co-design for cost-effective decoding, providing production context for StepMesh's role[9]. The vLLM project posted an RFC proposing ATTN-FFN disaggregation for MoE models[10][11], and a dedicated challenges paper appeared at arXiv 2602.09721[12]. Formal research has addressed both analytical provisioning under stochastic workloads[13] and the hardware systems challenges the approach surfaces for modern MoE architectures[14]. MegaScale-Infer, described as slashing LLM serving costs for MoE models through disaggregation[15], represents another production system in the same architectural direction. Financial commentators picked up PD disaggregation as evidence for extended GPU hardware ROI, arguing that older prefill-heavy GPUs can be repurposed into disaggregated decode pools[16][17]. Practitioner consensus has hardened around co-located prefill and decode at scale being actively wasteful[18][19], though vLLM's own documentation still labels disaggregated prefilling as experimental[20].

Two independent research efforts have targeted the same structural inefficiency in reinforcement learning post-training: the long-tail distribution of rollout lengths that creates GPU bubbles and degrades training throughput. Distribution-Aware Speculative Decoding (DASD), from Together.ai and WukLab (arXiv 2511.13841), adapts draft model behavior to the actual rollout length distribution and was presented at MLSys 2026, with Together.ai claiming up to 50% acceleration of RL rollouts[21][22]. Separately, MIT HAN Lab's Adaptive Drafter paper (arXiv 2511.16665), appearing at ASPLOS'26 under the open-source repository mit-han-lab/fastrl[23], has received MIT News coverage framing it as capable of substantially increasing LLM training efficiency[24], with at least one secondary source characterizing the speedup as 2x[25]. A FAISys 2025 paper also tackled dynamic speculative decoding for RL[26]. The simultaneous emergence of multiple independent approaches targeting the same long-tail RL inefficiency — from different institutions and different conference venues — signals that this is a high-value, recognized problem area in systems research, though direct comparisons between the approaches have not yet appeared.

For mixture-of-experts (MoE) models — now dominant at the frontier — expert load balancing at serving time remains a significant underexplored challenge in open source. SemiAnalysis noted that this problem surfaces primarily in closed production systems and receives disproportionately little open-source attention[27]. A cross-layer load balancing paper for distributed MoE inference appeared at ICML 2026[28], and load-balancing losses applied during MoE training complicate runtime scheduling decisions[29]. MegaScale-Infer's claimed cost reductions for MoE serving via disaggregation[15] are notable in this context, though whether the approach specifically addresses expert balancing has not been publicly detailed. Industry presence at MLSys 2026 was broad: vLLM, LMSYS, Inferact (co-hosting a luncheon with a16z), and Delta Institute were among the organizations attending[30][31][32][33].