Version 4

Agentic AI workloads are generating token volumes that invalidate prior infrastructure planning — empirical analysis of 432k coding-agent requests finds a median input of 96k tokens with roughly half exceeding 128k.[1][2] An independent academic study corroborates the same question from a research angle, analyzing how coding agents allocate token spend across task steps.[3] The engineering response has expanded to three documented technical pillars: tiered KV cache architectures (including new SSD-backed approaches[8]), IO-aware attention optimization, and speculative decoding, which uses a small draft model to propose tokens that a larger model verifies in parallel — with practitioner-reported throughput gains up to 3×.[20] At the competitive level, reports of NVIDIA acquiring Groq[22] — weeks after Cerebras priced its IPO[23] — reshapes the fast-inference hardware landscape, while enterprise analysts have begun quantifying an 'agentic sticker shock' problem: per-token prices are projected to fall 90% by 2030 per Gartner, yet total AI bills keep rising as usage scales.[29]

The gap between falling per-token prices and rising total enterprise AI spend is the defining economic tension of the agentic era. Which combination of technical mitigations — KV cache management, attention optimization, speculative decoding, and model-tier selection — closes that gap will determine whether AI agents achieve broad economic viability or remain confined to high-margin enterprise buyers who can absorb the 'sticker shock.'

Empirical analysis of 432k coding-agent requests by SemiAnalysis found the median input token count is 96k — roughly the length of a novel — with approximately half of all requests exceeding 128k tokens.[1][2] An independently published academic paper at OpenReview examines the same question from a research angle, studying how coding agents allocate token expenditure across task steps and attempting to build predictive models for consumption.[3] The structural driver is not user verbosity but the scaffolding agents assemble automatically before a user types a word: system prompts, tool and skill definitions, MCP schemas, and the rolling context of prior conversation turns and file contents agents must carry to stay coherent. A community observer responding to the SemiAnalysis data captured the dynamic precisely: "the 'task' is often a giant read of the codebase before any actual work begins."[4] A separate observer framed it as a memory problem the market systematically underprices.[5]

The infrastructure response to this token volume has matured into three distinct technical pillars. The first is tiered KV cache management: when a model processes a 96k-token request, the key-value tensors overflow GPU HBM at scale,[6] and the mitigation is spilling to slower memory tiers. NVIDIA's Dynamo framework productizes hierarchical offloading (HBM → DRAM → NVMe),[7] while a new paper titled Tutti documents making SSD-backed KV cache practical for long-context serving with detailed latency-throughput tradeoffs.[8] Dell has also published an enterprise storage approach to the same KV offloading problem.[9] A Substack post titled "The KV Cache Wars?" explicitly frames the competition between these approaches as an active industry battleground.[10] Open-source stacks including vLLM, SGLang, and lmdeploy are implementing KV cache quantization in INT4, INT8, and FP8 formats to reduce memory footprints.[11][12][13] The second pillar is IO-aware attention optimization: at long contexts, attention computation — not just storage — becomes a primary cost driver, and algorithms like FlashAttention restructure computation to minimize HBM reads/writes, with 1.4–1.7× wall-clock speedups observed at 98k tokens.[14][15] The third pillar, which has seen a burst of practitioner documentation, is speculative decoding: a small draft model proposes several tokens that the full model verifies in parallel rather than autoregressively, yielding throughput gains. Together.ai, NVIDIA, AWS, Snowflake, and BentoML have all published deployment-oriented guides targeting long-context and decode-heavy workloads,[16][17][18][19][20] and Nebius specifically examines how speculative decoding addresses latency budgets for large MoE models whose size otherwise breaks interactive latency targets.[21]

At the competitive level, two significant shifts have emerged around fast-inference hardware. Reports surfaced in a Reddit thread that NVIDIA has acquired Groq,[22] one of Cerebras's primary rivals in the specialist inference silicon market. If confirmed, this would represent NVIDIA absorbing a key independent challenger in the same space Cerebras entered public markets to pursue — Cerebras priced its IPO in May 2026 and NextPlatform covered its post-IPO strategy as a return to pushing inference performance limits independent of NVIDIA's ecosystem.[23] Meanwhile, GitHub Copilot's retirement of flat annual plans in favor of usage-based billing[24] has been followed by broader enterprise analyst attention to the same dynamic: HFS Research, CX Today, Galileo, and Quantiphi have each published analyses naming an 'agentic sticker shock' problem — with Galileo reporting that 40% of agentic AI projects fail before reaching production, due in part to cost overruns that were not anticipated at the pilot stage.[25][26][27][28]

The macro-level tension is what one community observer called 'the math looking broken': per-token prices are falling — Gartner projects a 90% reduction by 2030 — but total enterprise AI bills keep rising as usage scales.[29] This is a Jevons paradox dynamic: cheaper inference per unit drives more agentic deployment, which drives higher total spend, which makes the cost problem more urgent even as the per-token headline number improves. SemiAnalysis argues model labs — not cloud hyperscalers or inference resellers — are best positioned to capture value from this transition because they control model weights and can set inference economics at the source.[30] The counter-thesis is that smaller models with structured guardrails can reach near-frontier agentic accuracy at far lower cost,[31] potentially bifurcating the market between frontier-model enterprise deployments and cost-optimized small-model pipelines. A separate dimension of the same debate concerns open-weight models: MIT Sloan has raised questions about why open models — despite well-documented cost and customization advantages — are not more widely adopted in production agentic deployments.[32]