LLM Inference Efficiency: Phase, Layer, and Time Splitting Strategies Driving Cost Compression

LLM inference is being carved up through three successive hardware-level splits: phase disaggregation (routing prefill and decode to separate GPU pools), layer disaggregation (routing attention and feed-forward computation to hardware matched to each operation's profile), and time-based interleaving (alternating workload slices across a shared chip set to fill idle periods). [1][2] SemiAnalysis framed this three-cut framework as the central story of MLSys 2026, arguing that each split recovers previously wasted compute utilization and lowers cost per token. [1] Production results support the research: Anyscale reports 67% cost savings with prefill-decode (PD) disaggregation using Ray and vLLM on AMD MI325X hardware, [5] and the Kubernetes-native llm-d stack claims up to 70% higher throughput through the same technique. [7] Separately, UCSD's Hao AI Lab released JetSpec, a speculative decoding system using causal parallel tree drafting that achieves 9.64x speedup on MATH-500 and approximately 1,000 tokens per second on a single B200 GPU. [12]

Each successive optimization reduces the effective cost per token at the hardware level rather than through model compression or quality trade-offs. Industry observers including SemiAnalysis argue that cheaper tokens historically expand total inference demand rather than erode revenue, and early evidence — AI quarterly revenue surpassing hardware depreciation costs for the first time [18] — tentatively supports this. If the pattern holds, the practical ceiling on AI deployment shifts from compute cost to software and latency optimization.

LLM inference has been progressively subdivided by hardware function, with each subdivision designed to eliminate the idle compute that results from mixing workloads with incompatible resource profiles. SemiAnalysis laid out a three-part framework as the organizing theme of MLSys 2026: a phase split routes prefill (compute-bound) and decode (memory-bandwidth-bound) to separate hardware pools suited to each; a layer split routes attention layers (memory-hungry, favoring HBM-rich GPUs) and feed-forward layers (compute-hungry, favoring SRAM-based silicon) to different chip types; and time-based interleaving alternates workload slices across a shared chip set to fill gaps that would otherwise sit idle. [1][2] The common logic across all three, per SemiAnalysis, is finding idle compute and filling it — each step lowering cost per token and, the argument goes, growing total demand rather than compressing margins.

Prefill-decode disaggregation is the most production-mature of the three techniques. The core problem it addresses is that prefill and decode compete for the same GPU when served together: prefill requires high compute throughput while decode is bottlenecked by memory bandwidth, so a unified deployment forces a sub-optimal compromise for both. [3][4] Routing each to a dedicated hardware pool eliminates that compromise at the cost of transferring the KV cache between pools — a real overhead that practitioners treat as a manageable trade-off. [4] Anyscale validated this at scale, reporting 67% cost savings with Ray and vLLM in a disaggregated configuration on AMD MI325X. [5] The Kubernetes-native llm-d stack reports up to 70% higher tokens per second through the same approach, adding intelligent routing and autoscaling on top of standard serving infrastructure. [6][7] UCSD's Hao AI Lab, whose DistServe system pioneered much of the theoretical work on maximizing goodput (tokens delivered within latency SLA, rather than raw throughput), has tracked adoption from initial proposal in 2024 through substantial production uptake by late 2025. [8][9] Red Hat has published a multi-part technical series targeting enterprise deployments, covering disaggregation alongside KV cache tiering and EAGLE speculative decoding. [10][11]

JetSpec, also from Hao AI Lab, represents a distinct optimization line: applying causal parallel tree drafting to speculative decoding. Speculative decoding uses a smaller draft model to propose candidate token sequences that a larger verifier then accepts or rejects in parallel — recovering throughput without changing output quality. JetSpec's contribution is co-optimizing draft cost and draft quality by exploring multiple candidate paths simultaneously rather than a single draft chain. Benchmarks show 9.64x end-to-end speedup on MATH-500 (a reasoning benchmark) and 4.58x on open-ended chat; with CUDA graph and kernel optimizations, the system reaches approximately 1,000 tokens per second on a single B200 GPU. [12][13] The team designed JetSpec for integration with vLLM and SGLang, though production integration is still forthcoming. [14] SemiAnalysis explicitly flagged this result alongside the three-cut macro narrative, treating speculative decoding as a fourth dimension of inference efficiency rather than a separate research track.

At the hardware level, a secondary debate concerns which silicon benefits most from these algorithmic advances. Anyscale's 67% savings figure is on AMD MI325X rather than NVIDIA hardware, and community discussion mentions AMD's Helios chip as a potential native GPU disaggregation solution. [5][15] AMD has received analyst upgrades to a $1T–$1.14T market cap range, partly on inference hardware positioning. [16] Meanwhile, Cerebras reports 750 tokens per second for its 5.6 model — a benchmark reflecting a different hardware approach (wafer-scale SRAM) that sidesteps the memory-bandwidth bottleneck that makes disaggregation necessary in the first place. [17] The convergence of algorithmic splitting strategies with a diversifying hardware market means that inference architecture decisions are increasingly hardware-specific rather than portable.