Version 2

Agentic AI workloads are consuming tokens at a scale that invalidates most prior infrastructure planning, with empirical data from 432k coding-agent requests showing a median input of 96k tokens and roughly half of all requests exceeding 128k tokens.[1][2] The driver is structural: agents assemble enormous prefill contexts from system prompts, tool definitions, MCP schemas, and conversation history before any user input.[1] The industry's engineering response is now multi-pronged — tiered KV cache memory architectures, open-source quantization (INT4/INT8/FP8) in vLLM, SGLang, and lmdeploy,[5][6][7] and prompt caching evaluated specifically for agentic tasks.[8][9] Capital markets are pricing in the hardware bet: Cerebras priced the biggest IPO of 2026 in May,[13] and Ben Thompson's Stratechery has entered the conversation with 'The Inference Shift.'[14]

The transition from flat-rate to usage-based AI pricing[12] is no longer a pricing philosophy debate — it is being enforced by the token math of agentic workloads. Which mitigation strategies (hardware acceleration, quantization, caching) win at scale will determine both which companies capture value from the agentic AI wave and whether broad consumer access to AI agents is economically viable at all, or remains confined to high-margin enterprise deployments.

A wave of empirical data is forcing a reassessment of how expensive agentic AI actually is to serve. SemiAnalysis, analyzing 432k coding-agent requests from its own production environment, found the median input token count is 96k — roughly the length of a full novel — with approximately half of all requests already exceeding 128k tokens.[1][2] The culprit is not verbose users but the scaffolding agents assemble before a user types a single word: system prompts, tool and skill definitions, MCP schemas, and the rolling context of prior conversation turns and file contents that agents must carry to remain coherent across steps.[1] This structural inflation in prefill context length is quiet, automatic, and compounding.

The infrastructure consequence is a KV cache storage crisis. When a model processes a 96k-token request, the key-value tensors it generates quickly overflow GPU high-bandwidth memory capacity at scale.[3] The engineering response is now multi-pronged. Tiered KV cache architectures — spilling from HBM into DRAM, NVMe, or network-attached storage — have been productized by NVIDIA through its Dynamo framework.[4] Open-source inference stacks including vLLM, SGLang, and lmdeploy are actively implementing KV cache quantization in INT4, INT8, and FP8 formats,[5][6][7] which can roughly halve the memory footprint of the cache at some precision cost. Prompt caching — storing and reusing computed key-value state for fixed context prefixes — has been formally evaluated for long-horizon agentic tasks[8] and shows dramatic cost reductions in practice: one team reported cutting LLM costs by 59%.[9] The catch is that agentic workloads, with their constantly mutating context windows, can drive cache hit rates low enough to undercut those savings.[10] Researchers have also identified the attention mechanism itself — not just storage — as a primary cost driver at long contexts, with 1.4–1.7× wall-clock speedups observed at 98k tokens when attention computation is optimized.[11]

The market is translating these infrastructure pressures into pricing and capital events. GitHub Copilot announced that it is retiring flat annual plans in favor of usage-based billing,[12] an explicit acknowledgment that flat-rate pricing is economically unsustainable as per-session token counts climb. Cerebras priced the biggest IPO of 2026 in May,[13] a milestone that signals the fast-inference hardware thesis has matured beyond venture circles into public equity markets. Ben Thompson's Stratechery published 'The Inference Shift,'[14] bringing mainstream tech-strategy framing to what had been primarily an infrastructure-engineering conversation. SemiAnalysis argues that model labs — not cloud hyperscalers or inference resellers — are positioned to capture disproportionate value from this transition because they control model weights and can set inference economics at the source.[15]

An underappreciated counter-thesis runs through the open-source community: smaller models with carefully designed guardrails can achieve near-frontier accuracy on agentic tasks at a fraction of the token cost,[16] potentially bifurcating the market between premium frontier-model deployments and cost-optimized small-model pipelines. Open-weight models like DeepSeek V4 are competing on quality at the longer contexts agentic workloads demand.[17] The question is whether any combination of hardware acceleration, quantization, tiered memory, and prompt caching can compress serving costs fast enough to make 96k-median agentic sessions broadly affordable — or whether the efficiency gains will primarily benefit high-margin enterprise buyers.