The Information Machine

LLM Inference Efficiency: Phase, Layer, and Time Splitting Strategies Driving Cost Compression

closed · v4 · 2026-07-08 · 126 items · history

What's new in v4

Two substantive new items this pass: a Nexus paper proposes proactive intra-GPU PD disaggregation — performing the prefill/decode split within a single GPU through temporal scheduling rather than across separate nodes — directly addressing Rompel's fleet-scale argument by eliminating the inter-node infrastructure requirement. [11] UCSD's Hao AI Lab has published an 18-month retrospective on disaggregated inference, marking a milestone assessment from the technique's originators. [9] The remaining new items (a Medium explainer, a startup lexicon entry, and a ScienceDirect survey) are background reference rather than substantive developments.

What

LLM inference is being subdivided through three hardware-level splits — prefill/decode phase separation, attention/FFN layer separation, and time-based interleaving — each designed to route workloads to hardware matched to their compute profiles. [1][2] Prefill-decode disaggregation is production-mature: Anyscale reports 67% cost savings on AMD MI325X [5] and llm-d claims 70% higher throughput. [6] A Nexus paper now proposes proactive intra-GPU disaggregation — performing the prefill/decode split within a single GPU through temporal scheduling rather than across separate physical nodes — potentially extending disaggregation benefits below fleet scale. [11] UCSD's Hao AI Lab has published an 18-month retrospective on PD disaggregation's trajectory from proposal to widespread production adoption. [9]

Why it matters

Each split recovers compute that would otherwise sit idle, lowering cost per token without model quality loss. The intra-GPU approach addresses what practitioners identified as disaggregation's main practical constraint — the fleet-scale requirement — and if it delivers comparable gains within a single GPU, the technique becomes viable for deployments too small to keep continuously-loaded split node pools.

Open questions

  • Does intra-GPU (Nexus) disaggregation deliver throughput gains comparable to inter-node PD disaggregation, or does operating within a single GPU limit the specialization benefit? [11]

  • Does attention-FFN disaggregation deliver its theoretical gains in production, or does inter-node KV transfer overhead neutralize the benefit outside large fleets? [20][10]

  • Will a unified aggregation/disaggregation architecture displace the binary choice between prefill-decode separated and co-located serving? [21]

  • Does demand expand as per-token costs fall, or will enterprise AI inference spending plateau regardless of unit economics? [1][22]

Narrative

LLM inference has been progressively subdivided by hardware function, with each subdivision designed to eliminate idle compute resulting from mixing workloads with incompatible resource profiles. SemiAnalysis framed this at MLSys 2026 as a three-part progression: a phase split routes prefill (compute-bound) and decode (memory-bandwidth-bound) to separate hardware pools; a layer split routes attention layers (memory-hungry) and feed-forward layers (compute-hungry) to chip types matched to each; and time-based interleaving alternates workload slices across shared hardware to fill gaps. [1][2] The common logic across all three is finding idle compute and filling it, with each step reducing cost per token.

Prefill-decode (PD) disaggregation is the most production-mature technique. The core inefficiency it addresses is that prefill and decode compete for the same GPU when co-located: prefill requires high compute throughput while decode is bottlenecked by memory bandwidth, forcing a sub-optimal compromise for both. [3][4] Routing each to dedicated hardware eliminates that compromise at the cost of transferring the KV cache between pools. [4] Anyscale reported 67% cost savings with Ray and vLLM on AMD MI325X [5]; llm-d claims up to 70% higher tokens per second through intelligent routing and autoscaling. [6] UCSD's Hao AI Lab, which proposed DistServe and established PD disaggregation's theoretical basis around maximizing goodput rather than raw throughput, has published an 18-month retrospective documenting the technique's adoption trajectory from proposal to widespread production use. [7][8][9] One practitioner (Rompel) argues disaggregation only pays at fleet scale, where enough sustained traffic keeps split prefill and decode pools continuously utilized. [10] A Nexus paper proposes proactive intra-GPU disaggregation as a response to this constraint: rather than separating prefill and decode across physical nodes, it performs the split within a single GPU through temporal scheduling, eliminating inter-node KV transfer overhead and potentially making the gains accessible at smaller deployment scales. [11]

Attention-FFN disaggregation (AFD) — the layer-level split — has moved from theoretical proposal to active research with production integration underway. Multiple systems have been published: AiDE targets cost-effective serving [12]; CMU work covers operator-level disaggregated serving [13]; researchers have derived theoretically optimal attention/FFN ratios for disaggregated configurations [14]; and a vLLM RFC tracks AFD implementation for mixture-of-experts models, where the attention/FFN ratio is already architecturally distinct. [15] The layer split targets a different inefficiency than the phase split: attention and FFN layers have different compute-to-memory ratios, so routing each to matched hardware avoids forcing both through a single GPU type well-suited to only one.

Beyond infrastructure-level disaggregation, efficiency is being pursued at multiple levels simultaneously. JetSpec applies causal parallel tree speculative decoding to reach 9.64x speedup on MATH-500 and approximately 1,000 tokens per second on a single B200 GPU, with vLLM and SGLang production integration pending. [16][17] A Grouped Query Experts paper finds that using only 9 of 16 query attention heads achieves 1.7–1.8x faster prefill on long-context inputs while matching baseline accuracy. [18] At the application layer, developers working with Fable 5 discovered that rendering text-heavy context as PNG images and sending them as vision inputs costs roughly 60% less than equivalent text tokens — a workaround requiring no infrastructure changes but introducing OCR-level accuracy loss on exact strings. [19]

Timeline

  • 2024-01-01: UCSD Hao AI Lab proposes DistServe, establishing PD disaggregation's theoretical basis around maximizing goodput rather than raw throughput. [7][8]
  • 2025-12-01: Prefill-decode disaggregation moves from early research to substantial production adoption. [32]
  • 2026-06-16: Anyscale publishes benchmark showing 67% cost savings with Ray+vLLM PD disaggregation on AMD MI325X. [5]
  • 2026-06-25: Industry commentary notes AI inference costs falling; developer shift toward price-competitive open-weight and Chinese models attributed to inference economics. [33][34]
  • 2026-06-26: Red Hat publishes distributed LLM inference series covering PD disaggregation, KV cache tiering, and EAGLE speculative decoding across two parts. [24][25][23]
  • 2026-06-27: Grouped Query Experts paper: 1.7–1.8x faster prefill using only 9 of 16 query attention heads, matching baseline accuracy on long-context inputs. [18]
  • 2026-06-29: llm-d Kubernetes-native inference stack highlighted for 70% throughput gains via PD disaggregation with intelligent routing and autoscaling. [26][6]
  • 2026-06-29: Practitioner thread (Prajjwal, nanoserve) explains the compute/bandwidth conflict in unified serving and the KV cache transfer cost of disaggregation as a real but manageable trade-off. [3][4]
  • 2026-06-30: JetSpec results amplified: 9.64x speedup on MATH-500, approximately 1,000 TPS on a single B200 GPU. [16]
  • 2026-07-01: SemiAnalysis publishes three-cut framework (phase, layer, time splitting) as the central inference-efficiency narrative from MLSys 2026. [1][2]
  • 2026-07-01: Meta paper finds aggressive quantization raises overthinking failures in reasoning models by up to 52%; a 50-token decoding penalty largely resolves it. [35]
  • 2026-07-01: AMD Helios proposed as a native GPU disaggregation solution; AMD receives analyst upgrades on inference hardware positioning. [29][30]
  • 2026-07-02: Practitioner (Rompel) argues disaggregation only pays at fleet scale, where traffic volume keeps split prefill and decode pools continuously utilized. [10]
  • 2026-07-02: Multiple AFD systems published: AiDE, CMU operator-level disaggregated serving, derivations of optimal attention/FFN ratios, and a vLLM RFC tracking production integration. [12][13][14][20][27][15]
  • 2026-07-04: Developers working with Fable 5 discover that rendering text as PNG images cuts inference costs by ~60% by exploiting fixed vision-token pricing; technique is lossy on exact strings and hashes. [19]
  • 2026-07-08: Nexus paper proposes proactive intra-GPU PD disaggregation, performing the prefill/decode split within a single GPU through temporal scheduling to eliminate inter-node KV transfer overhead. [11]
  • 2026-07-08: UCSD Hao AI Lab publishes 18-month retrospective on disaggregated inference, reviewing PD disaggregation's trajectory from DistServe proposal to production adoption. [9]

Perspectives

SemiAnalysis

Frames phase, layer, and time splitting as a unified progression recovering wasted hardware utilization; argues cheaper tokens grow total AI demand rather than compress revenue.

Evolution: Consistent analytical and demand-bullish position across JetSpec amplification and the three-cut macro thread.

[1][2][16]

UCSD Hao AI Lab

Originators of DistServe (PD disaggregation, optimizing goodput over raw throughput) and JetSpec (causal parallel tree speculative decoding); their 18-month retrospective confirms PD disaggregation has reached substantial production adoption since 2024.

Evolution: 18-month retrospective added this pass; analytical stance consistent.

[7][8][17][16][9]

Anyscale / Red Hat

Anyscale reports 67% cost savings with Ray+vLLM on AMD MI325X; Red Hat treats disaggregation, KV cache tiering, and speculative decoding as production-ready enterprise techniques.

Evolution: Consistent.

[5][23][24][25]

Open-source ecosystem (llm-d, vLLM, Ray)

Kubernetes-native stacks are productizing PD disaggregation with intelligent routing and autoscaling; vLLM's RFC for attention-FFN disaggregation signals AFD is next in the production integration pipeline.

Evolution: Consistent across PD disaggregation and AFD RFC coverage.

[6][26][15][7]

Rompel (practitioner)

Disaggregation only pays at fleet scale: split prefill/decode plus KV-aware routing requires enough sustained traffic to keep both pools continuously utilized; smaller deployments may not break even on the infrastructure cost.

Evolution: Consistent; the Nexus intra-GPU approach is a direct response to this concern.

[10]

Nexus paper authors

Proactive intra-GPU disaggregation can deliver PD disaggregation benefits within a single GPU via temporal scheduling, avoiding the inter-node KV transfer overhead and fleet-scale infrastructure requirements of traditional approaches.

Evolution: New voice this pass.

[11]

AFD research community (AiDE, CMU, academic papers)

Attention-FFN disaggregation is theoretically grounded and implementable; routing attention and FFN layers to hardware matched to each operation's profile improves efficiency, with optimal ratios derivable analytically.

Evolution: Consistent since first appearance; growing publication count.

[12][13][14][20][27][15]

Application-layer practitioners (pxpipe / Fable 5 developers)

Cost pressure is intense enough to motivate unconventional workarounds: rendering text as vision inputs exploits fixed image-token pricing to reduce cost roughly 60% for dense text workloads, without infrastructure changes — at the cost of OCR-level accuracy on exact strings.

Evolution: Consistent since first appearance.

[19]

Tensions

  • Fleet-scale requirement vs. claimed savings: Rompel argues disaggregation only pays when traffic is sufficient to keep split pools utilized, while Anyscale and llm-d report 67–70% savings without specifying a minimum scale threshold; the Nexus intra-GPU approach is a direct response to Rompel's concern. [10][5][6][11]
  • Intra-GPU vs. inter-node disaggregation: Nexus proposes performing PD disaggregation within a single GPU through temporal scheduling, while the mainstream approach routes prefill and decode to separate physical hardware pools; whether intra-GPU delivers comparable gains is unresolved. [11][5][6]
  • Disaggregation vs. unified architecture: a recent arXiv paper argues prefill-decode aggregation and disaggregation should be unified rather than treated as a binary choice, while production deployments treat disaggregation as the clear efficiency winner. [21][5][6]
  • Cost compression vs. demand expansion: SemiAnalysis argues falling per-token costs grow total AI demand; some market observers treat the same trend as deflationary pressure on AI revenue. [1][22][28]
  • NVIDIA vs. AMD for disaggregated inference: Anyscale's 67% savings result is on AMD MI325X and AMD Helios is proposed as a native disaggregation solution, directly contesting NVIDIA's assumed inference hardware dominance. [5][29][30]
  • Throughput vs. goodput as the right optimization target: DistServe's framing argues maximizing raw throughput ignores latency SLAs; the correct metric is goodput (tokens delivered within SLA), which changes how disaggregation trade-offs are evaluated. [7][8][31]

Status: active and growing

Sources

  1. [1] Inference keeps getting carved up, and every cut makes intelligence cheaper. — SemiAnalysis Twitter (2026-07-01)
  2. [2] A quick map of the three cuts. — SemiAnalysis Twitter (2026-07-01)
  3. [3] Prefill and decode fight for the same GPU. Prefill is compute bound, decode is memory bandwidth bound. Run them in the s... — reactive:inference-cost-optimization (2026-06-29)
  4. [4] Disaggregation goes further: prefill on one GPU pool, decode on another, ship the KV cache between them. You pay a trans... — reactive:inference-cost-optimization (2026-06-29)
  5. [5] 67% Cost Savings with PD Disaggregation Using Ray and vLLM on AMD MI325X — reactive:inference-cost-optimization (2026-06-16)
  6. [6] @GithubProjects Prefill/decode disaggregation yielding up to 70% higher tokens/sec is massive for scaling large models w... — reactive:inference-cost-optimization (2026-06-29)
  7. [7] [2401.09670] DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — reactive:mlsys-2026-inference-systems
  8. [8] Throughput is Not All You Need: Maximizing Goodput in LLM Serving using Prefill-Decode Disaggregation | Hao AI Lab @ UCSD — reactive:inference-cost-optimization
  9. [9] Disaggregated Inference: 18 Months Later | Hao AI Lab @ UCSD — reactive:inference-cost-optimization
  10. [10] @hasantoxr Disaggregation only pays at fleet scale. Split prefill/decode + KV-aware routing needs enough traffic to keep... — reactive:inference-cost-optimization (2026-07-02)
  11. [11] Nexus: Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving — reactive:inference-cost-optimization
  12. [12] AiDE: Attention-FFN Disaggregated Execution for Cost-Effective ... — reactive:inference-cost-optimization
  13. [13] CMU CSD PhD Blog - Operator-Level Disaggregated Serving for Efficient LLM Inference — reactive:inference-cost-optimization
  14. [14] Theoretically Optimal Attention/FFN Ratios in Disaggregated LLM ... — reactive:inference-cost-optimization
  15. [15] [RFC]: ATTN-FFN Disaggregation for MoE Models #22799 - GitHub — reactive:mlsys-2026-inference-systems
  16. [16] Parallel draft tree, tree-causal verification — SemiAnalysis Twitter (2026-06-30)
  17. [17] GitHub - hao-ai-lab/JetSpec: JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Causal Parallel Tree Drafting · GitHub — reactive:inference-cost-optimization
  18. [18] This paper makes long-context attention cheaper and faster by letting each token use only the query heads it needs. — Rohan Paul Twitter (2026-06-27)
  19. [19] Developers found a cheaper way to feed Fable 5 large context by showing it pictures of text. — Rohan Paul Twitter (2026-07-04)
  20. [20] How Far Can Disaggregation Go? A Design-Space Exploration of ... — reactive:inference-cost-optimization
  21. [21] Prefill-Decode Aggregation or Disaggregation? Unifying Both ... - arXiv — reactive:inference-cost-optimization
  22. [22] AI quarterly revenue surpasses depreciation costs for the first time—has the trillion-dollar bet on computing power ente... — reactive:inference-cost-optimization (2026-06-26)
  23. [23] Part 2 of our 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗔𝗜 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 series is now live on Red Hat Developer: 𝘖𝘱𝘵𝘪𝘮𝘪𝘻𝘪𝘯𝘨 𝘋𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘈𝘐 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦: 𝘈𝘥𝘷... — reactive:inference-cost-optimization (2026-06-30)
  24. [24] Red Hat posted a technical article on distributed LLM inference: prefill/decode disaggregation, KV cache tiering, EAGLE ... — reactive:inference-cost-optimization (2026-06-27)
  25. [25] Scale your distributed #AI inference from a single vLLM instance to a multi-tenant grid. Explore prefill-decode disaggre... — reactive:inference-cost-optimization (2026-06-26)
  26. [26] llm-d is a Kubernetes-native inference serving stack that adds intelligent routing, KV-cache management, and autoscaling... — reactive:inference-cost-optimization (2026-06-29)
  27. [27] Attention-FFN disaggregation — reactive:mlsys-2026-inference-systems
  28. [28] OpenAI and Anthropic love to compare AI to electricity. 'AI will be like a utility,' they say. 'Infrastructure for the n... — reactive:inference-cost-optimization (2026-06-30)
  29. [29] @MilkRoadAI $AMD will solve this issue with Helios by providing for native GPU disaggregation. — reactive:inference-cost-optimization (2026-07-01)
  30. [30] $AMD is now re-rated to $1T - $1.14T market cap from different analysts/researched firms🧵 — reactive:inference-cost-optimization (2026-06-29)
  31. [31] Compute-bound and memory-bandwidth-bound are not the same problem. Treating them like one is costing you throughput. — reactive:inference-cost-optimization (2026-06-30)
  32. [32] UC San Diego Hao AI Labのブログ非常に分かりやすくDistServeのPrefill Decode Disaggregationの有用性について解説している上に、DistServeを提案した2024年から2025年末ま... — reactive:inference-cost-optimization (2026-06-27)
  33. [33] The cost of running an AI model dropped again this week, and the trend is the whole story. — reactive:inference-cost-optimization (2026-06-25)
  34. [34] The shift toward Chinese/open-weight models was already happening because developers follow price, latency, availability... — reactive:gpt-56-launch-government-access (2026-06-26)
  35. [35] Paper from Meta shows Quantized reasoning models often lose because they keep doubting a correct answer instead of finis… — Rohan Paul Twitter (2026-07-01)