Version 2

LLM inference is being subdivided through three successive hardware-level splits: prefill-decode disaggregation (routing compute-bound prefill and bandwidth-bound decode to separate hardware pools), attention-FFN disaggregation (routing attention and feed-forward layers to hardware matched to each operation's memory profile), and time-based interleaving. [1][2] Prefill-decode disaggregation is production-mature, with Anyscale reporting 67% cost savings on AMD MI325X [5] and llm-d claiming 70% higher throughput. [6] Attention-FFN disaggregation (AFD) has moved from theoretical proposal to active research, with multiple named systems — AiDE [10], CMU operator-level disaggregated serving [11] — and a vLLM production RFC now tracking implementation. [14] A practical constraint has emerged: at least one practitioner argues disaggregation only pays at fleet scale, where traffic is sufficient to keep split prefill and decode pools both continuously utilized. [7]

Each split recovers compute that would otherwise sit idle, lowering cost per token without sacrificing model quality. The layer-level AFD split is now close enough to production that its feasibility at typical deployment scales — not just research benchmarks — will determine whether it joins PD disaggregation as a standard serving technique or remains confined to large-fleet operators.

LLM inference has been progressively subdivided by hardware function, with each subdivision designed to eliminate idle compute that results from mixing workloads with incompatible resource profiles. SemiAnalysis laid out a three-part framework as the organizing theme of MLSys 2026: a phase split routes prefill (compute-bound) and decode (memory-bandwidth-bound) to separate hardware pools suited to each; a layer split routes attention layers (memory-hungry, favoring HBM-rich GPUs) and feed-forward layers (compute-hungry, favoring SRAM-based silicon) to different chip types; and time-based interleaving alternates workload slices across a shared chip set to fill gaps that would otherwise sit idle. [1][2] The common logic across all three, per SemiAnalysis, is finding idle compute and filling it — each step lowering cost per token and, the argument goes, growing total demand rather than compressing margins.

Prefill-decode (PD) disaggregation is the most production-mature technique. The core problem it addresses is that prefill and decode compete for the same GPU when co-located: prefill requires high compute throughput while decode is bottlenecked by memory bandwidth, so a unified deployment forces a sub-optimal compromise for both. [3][4] Routing each to a dedicated hardware pool eliminates that compromise at the cost of transferring the KV cache between pools — a real overhead practitioners treat as manageable. [4] Anyscale reported 67% cost savings with Ray and vLLM on AMD MI325X [5]; the Kubernetes-native llm-d stack claims up to 70% higher tokens per second through the same approach, adding intelligent routing and autoscaling on top of standard serving infrastructure. [6] However, at least one practitioner (Rompel) argues that disaggregation only pays at fleet scale, where enough traffic keeps split prefill and decode pools both continuously utilized; KV-aware routing at lower traffic volumes does not justify the infrastructure overhead. [7] UCSD's Hao AI Lab, whose DistServe paper established the theoretical basis for PD disaggregation around maximizing goodput (tokens delivered within latency SLA) rather than raw throughput, tracked adoption from initial proposal in 2024 through substantial production uptake by late 2025. [8][9]

Attention-FFN disaggregation (AFD) — the layer-level split in the three-cut framework — has moved from theoretical proposal to active research and early production planning. Multiple systems have been published: AiDE (Attention-FFN Disaggregated Execution) targets cost-effective serving [10]; a CMU PhD blog covers operator-level disaggregated serving [11]; researchers have derived theoretically optimal attention/FFN ratios for disaggregated configurations [12]; and a design-space exploration paper examines how far disaggregation can be extended beyond the phase level. [13] On the open-source side, a vLLM RFC tracks attention-FFN disaggregation for mixture-of-experts models specifically, where the attention/FFN ratio is already architecturally distinct. [14] Glenn Klockwood's technical notes document the AFD concept and implementation considerations. [15] The layer split targets a different inefficiency than the phase split: in dense models and especially MoE architectures, attention and FFN layers have different compute-to-memory ratios, so routing them to hardware optimized for each avoids forcing both through a single GPU type that is well-matched to only one.

Three further efficiency angles run parallel to the disaggregation story. JetSpec, from Hao AI Lab, applies causal parallel tree speculative decoding to reach 9.64x speedup on MATH-500 and approximately 1,000 tokens per second on a single B200 GPU, with vLLM and SGLang production integration still forthcoming. [16][17] A paper on Grouped Query Experts finds that attention can be made sparse within grouped-query attention — each token using only 9 of 16 query heads — achieving 1.7–1.8x faster prefill for long-context inputs while matching baseline accuracy. [18] And a Meta paper on quantized reasoning models finds that aggressive quantization raises overthinking failures (second-guessing already-correct answers) by up to 52% across benchmarks, though a lightweight fix — a decoding penalty on 50 hesitation tokens — reduces reasoning chain length by 12–23% while maintaining accuracy. [19] These results collectively show that inference efficiency is being pursued at multiple levels simultaneously: hardware routing, speculative decoding, attention sparsification, and decode-time behavioral correction.