Version 3

Agentic AI workloads are consuming tokens at a scale that invalidates most prior infrastructure planning, with empirical data from 432k coding-agent requests showing a median input of 96k tokens and roughly half of all requests exceeding 128k tokens.[1][2] The driver is structural: agents assemble enormous prefill contexts from system prompts, tool definitions, MCP schemas, and conversation history before any user types a word.[1] The industry's engineering response spans tiered KV cache architectures, open-source quantization (INT4/INT8/FP8) in vLLM, SGLang, and lmdeploy,[5][6][7] IO-aware attention algorithms like FlashAttention that reduce HBM bandwidth demands at long context,[8][9] and prompt caching evaluated specifically for agentic workloads.[22][11] Capital markets are pricing in the hardware bet: Cerebras priced a widely-covered IPO in May 2026,[14][15] and Ben Thompson's Stratechery framed the shift as a structural market transition with 'The Inference Shift.'[18]

The transition from flat-rate to usage-based AI pricing[13] is no longer a pricing philosophy debate — it is being enforced by the token math of agentic workloads. Which mitigation strategies (hardware acceleration, quantization, caching, attention optimization) win at scale will determine both which companies capture value from the agentic AI wave and whether broad access to AI agents is economically viable at all, or remains confined to high-margin enterprise deployments.

A wave of empirical data is forcing a reassessment of how expensive agentic AI is to serve. SemiAnalysis, analyzing 432k coding-agent requests from its own production environment, found the median input token count is 96k — roughly the length of a full novel — with approximately half of all requests already exceeding 128k tokens.[1][2] The culprit is not verbose users but the scaffolding agents assemble before a user types a single word: system prompts, tool and skill definitions, MCP schemas, and the rolling context of prior conversation turns and file contents that agents must carry to remain coherent across steps.[1] This structural inflation in prefill context length is quiet, automatic, and compounding.

The infrastructure consequence is a KV cache storage crisis. When a model processes a 96k-token request, the key-value tensors it generates quickly overflow GPU high-bandwidth memory capacity at scale.[3] The engineering response is now multi-pronged. Tiered KV cache architectures — spilling from HBM into DRAM, NVMe, or network-attached storage — have been productized by NVIDIA through its Dynamo framework.[4] Open-source inference stacks including vLLM, SGLang, and lmdeploy are actively implementing KV cache quantization in INT4, INT8, and FP8 formats,[5][6][7] which can roughly halve the memory footprint of the cache at some precision cost. Independently, the attention mechanism itself — not just storage — has been identified as a primary cost driver at long contexts, with IO-aware algorithms like FlashAttention restructuring attention computation to minimize HBM reads/writes,[8][9] yielding 1.4–1.7× wall-clock speedups observed at 98k tokens.[10] Prompt caching — storing and reusing computed key-value state for fixed context prefixes — shows dramatic cost reductions in practice: one team reported cutting LLM costs by 59%.[11] The catch is that agentic workloads, with their constantly mutating context windows, can drive cache hit rates low enough to undercut those savings.[12]

The market is translating these infrastructure pressures into pricing and capital events. GitHub Copilot announced the retirement of flat annual plans in favor of usage-based billing,[13] an explicit acknowledgment that flat-rate pricing is economically unsustainable as per-session token counts climb. Cerebras priced its IPO in May 2026, drawing coverage from CNBC,[14] Yahoo Finance,[15] and financial analysis outlets[16][17] as a signal that the fast-inference hardware thesis has matured beyond venture circles into public equity markets. Ben Thompson's Stratechery published 'The Inference Shift,'[18] bringing mainstream tech-strategy framing to what had been primarily an infrastructure-engineering conversation. SemiAnalysis argues that model labs — not cloud hyperscalers or inference resellers — are positioned to capture disproportionate value from this transition because they control model weights and can set inference economics at the source.[19]

An underappreciated counter-thesis runs through the open-source community: smaller models with carefully designed guardrails can achieve near-frontier accuracy on agentic tasks at a fraction of the token cost,[20] potentially bifurcating the market between premium frontier-model deployments and cost-optimized small-model pipelines. Open-weight models like DeepSeek V4 are competing on quality at the longer contexts agentic workloads demand.[21] The question is whether any combination of hardware acceleration, quantization, tiered memory, attention optimization, and prompt caching can compress serving costs fast enough to make 96k-median agentic sessions broadly affordable — or whether the efficiency gains will primarily benefit high-margin enterprise buyers.