Version 2

LLM inference efficiency research in mid-2026 has expanded from two core bottlenecks to three interrelated threads: the prefill/decode compute asymmetry [1], MoE expert-compute waste [2], and KV cache memory pressure [3][4]. Disaggregated prefill/decode architectures are advancing from theory to production-benchmarked reality, with vLLM/NIXL evaluations providing concrete throughput data [6] and cloud providers documenting approximately 2x throughput improvements [5]. KV cache quantization has crystallized as a distinct optimization track alongside sparse attention and expert pruning, with NVIDIA's NVF P4 approach targeting long-context workloads [8] and KVQuant pushing toward 10-million-token context via aggressive KV compression [4].

These three efficiency threads—disaggregated serving, expert pruning, and KV cache compression—target different resource constraints (compute scheduling, routing waste, and memory capacity) and are in principle additive. If they compose well in production stacks, cumulative gains could substantially reshape the economics of frontier model serving. The field is now moving from demonstrating individual gains to the harder question of whether these optimizations interact well under real workload diversity.

LLM inference in 2026 is being shaped by three converging bottleneck threads, each targeting a distinct layer of compute or memory waste. The first concerns the structural asymmetry between prefill (processing input prompts, compute-bound) and decode (generating output tokens, memory-bandwidth-bound) [1]. The second targets Mixture-of-Experts architectures, where research claims a large fraction of expert routing computation is removable without accuracy loss [2]. The third—now crystallizing into its own research cluster—addresses KV cache memory pressure: as context windows stretch toward millions of tokens, the key-value tensors produced by attention mechanisms become a dominant memory bottleneck, constraining both throughput and achievable context length [3][4].

The prefill/decode disaggregation approach—physically separating the two computation phases onto different hardware pools—has progressed from architectural proposal to production-facing evaluation. Cloud infrastructure provider Spheron documents approximately 2x throughput gains from disaggregated GPU deployments [5], while a community benchmark of disaggregated prefill/decode in vLLM using NIXL (NVIDIA's cross-architecture data-transfer library) provides more granular performance data for practitioners weighing deployment decisions [6]. This shift from theoretical to empirical matters: disaggregation was previously described as a promising architectural direction, but benchmarks in widely-deployed serving frameworks make tradeoffs legible to operators evaluating whether infrastructure overhead is worth the throughput gain. NVIDIA continues to invest in both sides of this picture, having previously advocated chunked prefill via TensorRT-LLM [7] and now publishing a complementary approach using NVF P4 quantization for KV cache storage, targeting long-context and large-batch workloads where KV cache size is the primary constraint [8].

KV cache quantization is emerging as a practical complement to compute-focused optimizations, targeting memory capacity rather than scheduling efficiency. The core approach compresses cached attention keys and values into lower-precision formats, reducing memory footprint and enabling longer effective context windows without architectural changes. NVIDIA's NVF P4 KV cache work targets production serving scenarios [8], while KVQuant (NeurIPS 2024) demonstrated a research path toward 10-million-token context via per-channel, per-vector quantization of KV tensors [4]. Community discussion reflects growing practitioner recognition that the KV cache—not raw compute—is increasingly the primary serving bottleneck in long-context deployments [3]. Unlike MoE expert pruning or sparse attention, which reduce compute, KV cache quantization directly addresses memory, making it additive in principle to those approaches. The Alibaba and Nanjing University sparse attention paper, which claimed a 9.36× speedup on million-token prefill [9], and the MoE expert pruning literature [2] together represent compute-side gains; KV cache quantization closes the memory side.

The broader serving optimization landscape includes two additional research directions. Speculative decoding—using a smaller draft model to propose tokens that a larger model verifies in parallel—now has 2026 production benchmark data [10], providing empirical grounding for a technique previously evaluated mainly in controlled settings. AdaServe, a CMU paper accepted to EuroSys 2026, addresses a different dimension: multi-SLO serving, adaptively allocating compute across requests with heterogeneous latency requirements [11]. The AdaServe framing implicitly challenges throughput-maximizing approaches like disaggregation by arguing that production fleets serve requests with fundamentally different latency tolerance, and that ignoring this diversity leaves significant quality-of-service gains on the table. Together, these results paint a serving infrastructure landscape where disaggregation, KV compression, sparse attention, expert pruning, speculative decoding, and SLO-aware scheduling are all active optimization levers—but the field has not yet demonstrated how these techniques compose in a single production stack.