Version 3

LLM inference is being subdivided through three hardware-level splits — prefill/decode phase separation, attention/FFN layer separation, and time-based interleaving — each designed to route workloads to hardware matched to their compute profiles. [1][2] Prefill-decode disaggregation is production-mature: Anyscale reports 67% cost savings on AMD MI325X [5] and llm-d claims 70% higher throughput. [6] Attention-FFN disaggregation has moved from theory to active research with multiple published systems and a vLLM RFC tracking implementation. [10][13] At the application layer, a separate cost-reduction technique has appeared: rendering text-heavy context as PNG images to exploit fixed vision-token pricing, achieving roughly 60% cost reduction on Fable 5. [17]

Each infrastructure split recovers compute that would otherwise sit idle, lowering cost per token without model quality loss. The emergence of creative application-layer hacks alongside infrastructure-level disaggregation shows cost pressure is intense enough to motivate unconventional approaches at every level of the stack. Whether attention-FFN disaggregation can match prefill-decode disaggregation's production uptake — and whether these gains grow total AI demand or compress provider margins — remains unresolved.

LLM inference has been progressively subdivided by hardware function, with each subdivision designed to eliminate idle compute resulting from mixing workloads with incompatible resource profiles. SemiAnalysis framed this as a three-part progression at MLSys 2026: a phase split routes prefill (compute-bound) and decode (memory-bandwidth-bound) to separate hardware pools; a layer split routes attention layers (memory-hungry) and feed-forward layers (compute-hungry) to chip types matched to each; and time-based interleaving alternates workload slices across shared hardware to fill gaps. [1][2] The common logic across all three is finding idle compute and filling it, with each step reducing cost per token.

Prefill-decode (PD) disaggregation is the most production-mature technique. The core inefficiency it addresses is that prefill and decode compete for the same GPU when co-located: prefill requires high compute throughput while decode is bottlenecked by memory bandwidth, forcing a sub-optimal compromise for both. [3][4] Routing each to dedicated hardware eliminates that compromise at the cost of transferring the KV cache between pools, an overhead practitioners treat as manageable. [4] Anyscale reported 67% cost savings with Ray and vLLM on AMD MI325X [5]; the Kubernetes-native llm-d stack claims up to 70% higher tokens per second through intelligent routing and autoscaling. [6] UCSD's Hao AI Lab, whose DistServe paper established the theoretical basis for PD disaggregation around maximizing goodput rather than raw throughput, tracked adoption from initial proposal in 2024 through substantial production uptake by late 2025. [7][8] One practitioner (Rompel) qualifies these savings: disaggregation only pays at fleet scale, where enough sustained traffic keeps split prefill and decode pools both continuously utilized. [9]

Attention-FFN disaggregation (AFD) — the layer-level split — has moved from theoretical proposal to active research with production integration underway. Multiple systems have been published: AiDE targets cost-effective serving [10]; CMU work covers operator-level disaggregated serving [11]; researchers have derived theoretically optimal attention/FFN ratios for disaggregated configurations [12]; and a vLLM RFC tracks AFD implementation for mixture-of-experts models, where the attention/FFN ratio is already architecturally distinct. [13] The layer split targets a different inefficiency than the phase split: attention and FFN layers have different compute-to-memory ratios, so routing each to matched hardware avoids forcing both through a single GPU type well-suited to only one.

Beyond infrastructure-level disaggregation, efficiency is being pursued at multiple levels simultaneously. JetSpec applies causal parallel tree speculative decoding to reach 9.64x speedup on MATH-500 and approximately 1,000 tokens per second on a single B200 GPU, with vLLM and SGLang production integration pending. [14][15] A Grouped Query Experts paper finds that using only 9 of 16 query heads achieves 1.7–1.8x faster prefill on long-context inputs while matching baseline accuracy. [16] At the application layer, developers working with Fable 5 discovered that rendering text-heavy context as PNG images and sending them as vision inputs costs roughly 60% less than equivalent text tokens: a 1928×1928 image costs approximately 4,761 vision tokens regardless of content density, whereas the same image can hold around 92,000 characters of dense code. [17] The approach is lossy — the model reads the image through OCR and may misread exact strings or hashes — but for use cases that tolerate that tradeoff, it is a substantial cost reduction achievable without any infrastructure changes.