Version 3

LLM inference efficiency research has consolidated around three bottleneck threads—disaggregated prefill/decode serving, MoE expert pruning, and KV cache memory pressure—while the KV cache thread is itself splitting into two parallel sub-tracks: quantization (compressing retained KV tensors to lower precision [4][3]) and eviction (selectively dropping entire tokens from the cache [5][6][7]). A nascent 'LLM tokenomics' framing now argues that token value is a function of embedded intelligence and delivery speed [12], beginning to connect infrastructure-layer efficiency research to user-perceived value.

Quantization and eviction attack the same KV cache bottleneck from different angles with different accuracy tradeoff profiles; if they compose well, their combination could unlock context lengths and serving densities beyond what either achieves alone. The tokenomics framing, however nascent, creates a vocabulary for translating infrastructure tradeoffs into economic terms—potentially grounding adaptive serving strategies like AdaServe in user-value rather than pure throughput metrics.

LLM inference efficiency research is structured around three converging bottleneck threads: the compute asymmetry between prefill (processing input prompts, compute-bound) and decode (generating tokens, memory-bandwidth-bound) [1]; Mixture-of-Experts routing waste, where researchers claim roughly half of expert compute is removable without accuracy loss [2]; and KV cache memory pressure, where key-value tensors produced by attention mechanisms become the dominant constraint as context windows stretch toward millions of tokens [3]. Each thread targets a different layer of the serving stack, and in principle their gains are additive.

The KV cache thread is the most active frontier and is now splitting into two parallel sub-tracks with distinct tradeoff profiles. Quantization compresses cached key-value tensors into lower-precision formats—retaining all tokens but introducing precision error—as demonstrated by NVIDIA's NVF P4 KV cache work targeting production long-context serving [4] and KVQuant's research toward 10-million-token inference [3]. Eviction takes the opposite approach: selectively dropping entire tokens from the KV cache to reduce its size while maintaining full precision for retained entries. Samsung Research's LookaheadKV proposes glimpsing ahead in the generation process to predict which KV entries will be needed before evicting them [5]; KeyDiff (NeurIPS 2025) identifies redundant entries by measuring similarity between keys, on the premise that similar keys carry overlapping information [6]; and a concurrent ArXiv paper frames eviction as a general mechanism for improving long-context performance across deployment settings [7]. These approaches could in principle be stacked—eviction reduces token count, quantization compresses the remainder—but their interaction under real workloads has not yet been benchmarked.

On the compute side, prefill/decode disaggregation has moved from architectural proposal to production-facing benchmark. Spheron documents approximately 2x throughput gains from disaggregated GPU deployments [8], while community benchmarks of vLLM with NIXL (NVIDIA's cross-architecture data-transfer library) provide practitioner-legible performance data [9]. The Alibaba/Nanjing University sparse attention work claims a 9.36× prefill speedup at million-token scale [10], and the MoE expert pruning literature argues these gains are applicable to already-deployed frontier models with near-zero accuracy cost [2]. Speculative decoding—using a smaller draft model to propose tokens that a larger model verifies in parallel—now has 2026 production benchmark data [11].

A conceptual layer is beginning to emerge above individual optimizations. Rohan Paul's 'LLM tokenomics' framing argues that token value is determined by two factors: the intelligence embedded in the token and how quickly it arrives [12]. This connects infrastructure tradeoffs to user-perceived value in a way that individual efficiency metrics alone do not capture. AdaServe (CMU, EuroSys 2026) makes a related but more concrete argument: production serving fleets must handle requests with heterogeneous latency tolerance, and throughput-maximizing approaches systematically underserve latency-sensitive workloads [13]. Both framings point toward a next-phase challenge—not just finding individual efficiency gains, but assembling serving systems that allocate compute according to what tokens are actually worth to specific users.