The Information Machine

LLM Inference Efficiency Research Cluster · history

Version 4

2026-05-27 18:44 UTC · 42 items

What

LLM inference efficiency research is structured around three converging bottleneck threads—disaggregated prefill/decode serving, MoE expert pruning, and KV cache memory pressure—while the KV cache thread has now split into three parallel sub-tracks: quantization (compressing retained tensors to lower precision [7][8]), eviction (selectively dropping tokens from the cache [9][12][13]), and low-rank decomposition (factorizing KV tensors to reduce their dimensionality [14]). A nascent 'LLM tokenomics' framing argues token value is a function of embedded intelligence and delivery speed [16], beginning to connect infrastructure efficiency research to user-perceived value. Hardware co-design—combining processing-near-memory (PNM) architectures with sparse attention—has emerged as a distinct approach to million-token inference alongside pure software retrofits [5].

Why it matters

Three KV cache compression approaches—eviction, quantization, and low-rank decomposition—now address the same bottleneck from distinct angles with different accuracy-tradeoff profiles; their potential composability could unlock context lengths and serving densities beyond what any single approach achieves alone. The emergence of PNM hardware co-design alongside software-only approaches sharpens a capital-allocation fork: operators must decide whether long-context efficiency requires new hardware or can be solved at the software layer on existing GPU clusters.

Open questions

  • Do eviction, quantization, and low-rank KV cache compression methods compose—enabling stacked memory savings—or do they compete for the same marginal memory budget, forcing operators to choose among them? [14][9][7]

  • Can PNM hardware co-design with sparse attention [5] outperform pure software sparse attention retrofits [4] at million-token scale, and if so, does this shift the software-vs.-hardware tradeoff calculus for operators?

  • How much accuracy degradation do eviction approaches like LookaheadKV [9] and KeyDiff [12] introduce on reasoning-intensive tasks at long contexts, where dropped tokens may carry disproportionate semantic weight?

  • Can the 'tokenomics' framing—token value as intelligence × speed [16]—provide actionable per-request compute allocation for SLO-adaptive systems like AdaServe [15], or is it too coarse a framework to drive concrete serving decisions?

Narrative

LLM inference efficiency research is organized around three distinct bottleneck layers. At the compute layer, the asymmetry between prefill (processing input prompts, compute-bound) and decode (generating tokens, memory-bandwidth-bound) has driven both architectural proposals and production deployments [1]. Spheron documents approximately 2× throughput gains from disaggregated GPU deployments [2], and vLLM with NIXL—NVIDIA's cross-architecture data-transfer library—now has practitioner-facing benchmarks [3]. The Alibaba/Nanjing University sparse attention work claims a 9.36× prefill speedup at million-token scale [4], while an IEEE Computer paper takes the hardware co-design route, combining processing-near-memory (PNM) architectures with sparse attention to enable multi-million-token inference without depending solely on GPU compute [5]. At the model layer, MoE expert pruning researchers argue that roughly half of expert compute is removable post-training with near-zero accuracy loss on Qwen3 and GLM architectures [6].

The KV cache memory layer is the most active research frontier and has internally differentiated into three sub-tracks. Quantization compresses cached key-value tensors into lower-precision formats—retaining all tokens but introducing precision error—as shown by NVIDIA's NVF P4 work targeting production long-context serving [7] and KVQuant's research toward 10-million-token inference [8]. Eviction takes the opposite approach: selectively dropping entire tokens from the KV cache to reduce its size while maintaining full precision for retained entries. Samsung Research's LookaheadKV glimpses ahead in the generation process to predict which KV entries will be needed before evicting them [9][10][11]; KeyDiff (NeurIPS 2025) identifies redundant entries by measuring similarity between keys [12]; and a concurrent ArXiv paper frames eviction as a general mechanism for improving long-context performance [13]. A third sub-track—low-rank decomposition, which factorizes KV tensors to reduce their dimensionality rather than dropping or compressing them—rounds out the compression landscape [14]. These approaches carry different accuracy-tradeoff profiles: eviction loses information for dropped tokens, quantization introduces precision error uniformly, and low-rank methods introduce approximation error across the factorized dimensions.

A conceptual layer is beginning to emerge above individual optimizations. AdaServe (CMU, EuroSys 2026) argues that real production serving fleets have heterogeneous SLO requirements and that throughput-maximizing approaches systematically underserve latency-sensitive workloads; SLO-adaptive compute allocation is therefore a distinct and necessary optimization layer [15]. Rohan Paul's 'LLM tokenomics' framing argues that token value is determined by two factors—the intelligence embedded in the token and how quickly it arrives [16]—connecting infrastructure tradeoffs to user-perceived value in a way that individual efficiency metrics alone cannot capture. Both framings point toward a next-phase challenge: not just finding individual efficiency gains, but assembling serving systems that allocate compute according to what tokens are actually worth to specific users.

Timeline

  • 2024: KVQuant paper (NeurIPS 2024) establishes academic baseline for extreme-context KV cache quantization, targeting 10-million-token inference [8]
  • 2025: PNM Meets Sparse Attention paper (IEEE Computer) demonstrates hardware co-design enabling multi-million-token inference via processing-near-memory architecture [5]
  • Late 2025: KeyDiff (NeurIPS 2025) introduces key-similarity-based KV cache eviction for resource-constrained long-context inference [12]
  • 2026-03-04: Speculative decoding production benchmark for 2026 published, providing empirical throughput data outside controlled evaluation settings [23]
  • 2026-04-29: MarkTechPost survey frames KV cache compression as a three-track problem—eviction, quantization, and low-rank decomposition—consolidating the field's taxonomy [14]
  • 2026-05-21: Rohan Paul introduces 'LLM tokenomics' framing: token value as a function of embedded intelligence and delivery speed [16]
  • 2026-05-23: Discussion notes AI cluster bottleneck has shifted away from raw FLOPS toward memory bandwidth [24]
  • 2026-05-24: MoE expert waste finding amplified: 50% of compute removable with near-zero accuracy loss on Qwen3 and GLM architectures [6]
  • 2026-05-24: Alibaba/Nanjing University sparse attention paper publicized: 9.36× prefill speedup vs FlashAttention-2 at million-token scale [4]
  • 2026-05-24: Prefill/decode compute asymmetry analysis relayed; NVIDIA structural advantage in compute-bound prefill highlighted as context windows grow [1]
  • 2026-05: NVIDIA publishes NVF P4 KV cache quantization targeting long-context and large-batch LLM serving [7]
  • 2026-05: Spheron documents approximately 2× throughput gain from prefill/decode disaggregation on GPU cloud [2]
  • 2026-05: vLLM/NIXL disaggregated prefill/decode benchmark published, providing production-facing performance data for practitioners [3]
  • 2026-05: AdaServe (CMU) multi-SLO LLM serving paper accepted to EuroSys 2026, proposing SLO-adaptive compute allocation across heterogeneous requests [15]
  • 2026-05: Samsung Research's LookaheadKV published and widely amplified: future-glimpsing to improve KV cache eviction decisions without generation overhead [9][10][25][11]
  • 2026-05: ArXiv paper frames KV cache eviction as a general mechanism for improving long-context performance across deployment settings [13]

Perspectives

Rohan Paul (@rohanpaul_ai)

Enthusiastic amplifier of efficiency research across MoE pruning, sparse attention, and prefill/decode asymmetry; also proposes a 'tokenomics' framework defining token value as the product of intelligence and delivery speed

Evolution: consistent; tokenomics framing introduced in prior pass remains the distinctive contribution

[6][4][1][16]

NVIDIA (TensorRT-LLM / developer blog / research)

Advocates compute-layer solutions (chunked prefill in TensorRT-LLM) and memory-layer solutions (NVF P4 KV cache quantization), positioning NVIDIA hardware and software as jointly addressing inference efficiency across multiple resource dimensions

Evolution: consistent

[17][7]

KV cache compression research community (NVIDIA, KVQuant, Samsung, KeyDiff, low-rank methods)

Argues KV cache memory pressure is the primary serving bottleneck in long-context deployments, addressable through quantization, eviction, or low-rank decomposition; these three sub-tracks have different accuracy-tradeoff profiles and may be complementary or composable

Evolution: expanded: low-rank decomposition added as a third sub-track alongside previously tracked quantization and eviction

[7][8][9][12][13][14]

Hardware co-design research community (PNM/IEEE Computer)

Argues that software-only sparse attention retrofits are insufficient for multi-million-token inference at scale, and that processing-near-memory hardware co-design is required to fully unlock long-context efficiency gains

Evolution: new voice this pass

[5]

MoE pruning research community

Collectively argues that post-training expert pruning can recover roughly 50% of MoE compute waste with negligible accuracy cost, applicable to already-deployed frontier models including Qwen3 and GLM

Evolution: consistent; secondary coverage of 'Not All Experts Are Equal' paper deepens but does not alter the core claim

[6][18][19][20][21][22]

Cloud/infrastructure practitioners (Spheron, vLLM/NIXL community)

Frame prefill/decode disaggregation as a practical, deployable strategy with measurable throughput gains; translate architectural research into operator-facing benchmarks and deployment guidance

Evolution: consistent

[2][3]

CMU AdaServe team

Argues production serving fleets have heterogeneous SLO requirements and that throughput-maximizing approaches systematically underserve latency-sensitive requests; SLO-adaptive compute allocation is a distinct and necessary optimization layer

Evolution: consistent

[15]

Alibaba / Nanjing University researchers

Claims selective sparse attention achieves dramatic prefill speedup (9.36×) with lightweight adaptation, positioning sparsity as a software retrofit to existing GPU stacks rather than an architectural overhaul

Evolution: consistent

[4]

Tensions

  • KV cache eviction vs. quantization vs. low-rank decomposition: three compression sub-tracks attack the same memory bottleneck with different accuracy-tradeoff profiles—eviction loses information for dropped tokens, quantization introduces uniform precision error, low-rank introduces approximation error across factorized dimensions—and their composability under real workloads remains untested. [9][12][7][8][14]
  • Software retrofit vs. hardware specialization: sparse attention and MoE pruning favor software-level retrofits to existing GPU stacks, while PNM co-design research and disaggregation benchmarks point toward specialized or heterogeneous hardware as the structural long-term solution—implying different capital allocation and vendor lock-in strategies. [4][6][2][3][5]
  • Throughput-maximizing vs. SLO-diversity serving: disaggregation research optimizes for aggregate throughput assuming relatively homogeneous workloads, while AdaServe argues production fleets require SLO-adaptive allocation and that throughput-first optimization systematically disadvantages latency-sensitive requests. [2][3][15]
  • 'Free efficiency' claim vs. evaluation skepticism: researchers claim 50% MoE expert pruning and 9.36× sparse attention speedups arrive with near-zero accuracy cost, but no adversarial benchmarks or out-of-distribution reasoning evaluations have publicly tested these claims. [6][4]
  • Tokenomics framing vs. concrete SLO metrics: the 'token value = intelligence × speed' framework [16] offers intuitive vocabulary for connecting efficiency to user value, but AdaServe's SLO-adaptive approach requires precise, per-request latency targets—it is unclear whether the tokenomics framing is actionable at that granularity. [16][15]

Sources

  1. [1] Chamath on all important “prefill” and “decode.” in AI compute. — Rohan Paul Twitter (2026-05-24)
  2. [2] Prefill-Decode Disaggregation on GPU Cloud: Split LLM Inference for 2x Throughput (2026 Guide) | Spheron Blog — reactive:llm-inference-efficiency
  3. [3] Benchmarking Disaggregated Prefill/Decode in vLLM Serving with NIXL : r/LocalLLaMA — reactive:llm-inference-efficiency
  4. [4] New Alibaba + Nanjing Univ paper claims million-token prefill can be sped up 9.36X (compared against FlashAttention-2) w… — Rohan Paul Twitter (2026-05-24)
  5. [5] PNM Meets Sparse Attention: Enabling Multi-Million Tokens ... — reactive:llm-inference-efficiency
  6. [6] A large MoE model may be wasting half its expert compute on tokens that barely need expert help. — Rohan Paul Twitter (2026-05-24)
  7. [7] Optimizing Inference for Long Context and Large Batch Sizes with ... — reactive:llm-inference-efficiency
  8. [8] KVQuant: Towards 10 Million Context Length LLM Inference with KV ... — reactive:llm-inference-efficiency
  9. [9] BLOG | Samsung Research — reactive:llm-inference-efficiency
  10. [10] LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing ... — reactive:llm-inference-efficiency
  11. [11] LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing ... — reactive:llm-inference-efficiency
  12. [12] NeurIPS Poster KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments — reactive:llm-inference-efficiency
  13. [13] Towards Improving Long-Context Performance with KV Cache Eviction — reactive:llm-inference-efficiency
  14. [14] Top 10 KV Cache Compression Techniques for LLM Inference: Reducing Memory Overhead Across Eviction, Quantization, and Low-Rank Methods - MarkTechPost — reactive:llm-inference-efficiency
  15. [15] [PDF] AdaServe: Accelerating Multi-SLO LLM Serving with SLO ... — reactive:llm-inference-efficiency
  16. [16] "Not all tokens are created equal, and there is a way to look at token value. There are two key factors that impact toke… — Rohan Paul Twitter (2026-05-21)
  17. [17] Streamlining AI Inference Performance and Deployment with NVIDIA ... — reactive:llm-inference-efficiency
  18. [18] MoE Pathfinder: Trajectory-driven Expert Pruning - Semantic Scholar — reactive:llm-inference-efficiency
  19. [19] moe-pruner: Democratizing DeepSeek-v3 with Expert Fusion - GitHub — reactive:llm-inference-efficiency
  20. [20] MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router — reactive:llm-inference-efficiency
  21. [21] Layer-adaptive Expert Pruning for Pre-Training of Mixture-of-Experts Large Language Models — reactive:llm-inference-efficiency
  22. [22] Efficient Expert Pruning and Skipping for Mixture-of ... - Moonlight — reactive:llm-inference-efficiency
  23. [23] LLM Serving: Speculative Decoding Production Benchmark 2026 | Chaos and Order — reactive:llm-inference-efficiency
  24. [24] In the past, the bottleneck of AI clusters was FLOPS. — reactive:llm-inference-efficiency (2026-05-23)
  25. [25] LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing ... — reactive:llm-inference-efficiency