Version 1

MLSys 2026, the ninth annual Conference on Machine Learning and Systems, is underway in Bellevue, Washington (week of May 18–22, 2026)[1][2]. The conference's inference track is organized around four converging themes: sparse attention reaching production deployment, prefill-decode disaggregation maturing into an industry standard, MoE expert balancing emerging as an underexplored production challenge, and long-tail rollout distributions degrading RL training throughput[4][7][15][18]. NVIDIA's BLASST paper won the MLSys 2026 Best Paper award[3], and @haoailab — credited with originating prefill-decode disaggregation — is presenting a new disaggregation technique extending that architecture[7].

The systems research at MLSys 2026 marks the industrialization of inference techniques that have been in academic papers for one to two years. Decisions made now — which sparse attention kernels survive production hardening, how disaggregated architectures are composed, how MoE scheduling is solved at scale — will shape AI serving infrastructure for the next hardware generation. The investor community's uptake of PD disaggregation as a GPU-longevity argument[13][14] shows these engineering choices carry real capital implications.

MLSys 2026, held in Bellevue, Washington, brings together researchers and engineers from major AI labs and infrastructure companies to address production systems challenges[1][2]. This year's program is distinguished by a cluster of inference-focused research reflecting a broader shift: the frontier of AI systems work has moved from model architecture to the infrastructure layer — how to serve large models faster, cheaper, and at greater scale.

The conference's most prominent individual recognition went to NVIDIA's BLASST paper (Dynamic Blocked Attention Sparsity via Softmax Thresholding), which won the MLSys 2026 Best Paper award[3]. BLASST addresses the core inefficiency of dense attention at long context lengths by dynamically identifying and skipping low-salience attention blocks at runtime using a softmax threshold gate. Its award reflects a wider trend SemiAnalysis documented in a pre-conference thread: sparse attention has crossed from academic benchmarks into live production deployments, with DeepSeek's Sparse Attention and NousResearch's Lighthouse Attention both cited as examples[4]. Multiple reference papers and guides on sparse attention patterns have proliferated alongside these deployments[5][6].

Prefill-decode (PD) disaggregation — routing the compute-bound prefill phase and the memory-bandwidth-bound decode phase to dedicated hardware pools — is now widely characterized as an industry standard. SemiAnalysis credits @haoailab (Hao Zhang's DistServe group) as the originator and notes the group is presenting a further disaggregation technique at MLSys 2026, with StepFun AI demonstrating attention-FFN disaggregation as an additional axis of workload specialization[7]. Production deployment activity is visible across the ecosystem: LMCache Lab released an async PDBackend making disaggregated serving more efficient[8]; Theta EdgeCloud tested PD disaggregation for large-scale LLM serving[9][10]; and the Atlas system was integrated into a PD disaggregation setup using heterogeneous accelerators[11]. Commentary from practitioners at the conference frames co-located prefill and decode at scale as actively wasteful[12], while financial commentators picked up the architecture as evidence that GPU useful lives extend well beyond skeptics' estimates — older prefill-heavy GPUs can be repurposed into disaggregated decode pools[13][14].

Two further research threads are active. For mixture-of-experts (MoE) models — now dominant at the frontier — expert load balancing at serving time is a significant engineering challenge that SemiAnalysis observes receives disproportionately little open-source attention because it primarily surfaces in closed production systems[15]. Load-balancing losses applied during MoE training encourage expert diversity, which complicates runtime scheduling decisions[16]; an ICML 2026 paper on Cross-Layer Load Balancing in Distributed MoE Inference addresses the problem from a different angle[17]. On the training side, RL post-training workloads are exposing a structural inefficiency: the long-tail distribution of rollout lengths creates GPU bubbles in training pipelines. A new Distribution-Aware Speculative Decoding paper specifically targets this problem, building on draft-model techniques including Eagle, MTP, and DFlash[18]. SnorkelAI presented related work on RLVR in low-data, low-compute settings at the conference[19]. Industry presence is broad: vLLM, LMSYS, Inferact (co-hosting a luncheon with a16z), and Delta Institute are among the organizations attending[20][21][22][23].