Version 1

Agentic AI workloads are consuming dramatically more tokens than the industry assumed, and SemiAnalysis has published empirical data to prove it.[1] Analysis of 432k real coding-agent requests shows a median input of 96k tokens — larger than the full text of The Great Gatsby — with roughly half of all requests already exceeding 128k tokens.[2][1] The driver is not users typing longer prompts: it is the prefill context agents assemble before the user types anything — system prompts, tool definitions, MCP schemas, skill libraries, and prior-turn histories.[2] This structural shift is pushing KV cache storage beyond GPU high-bandwidth memory capacity and forcing infrastructure providers, model labs, and billing teams to respond.[3][8]

If the median coding-agent request already rivals a short novel in token count, inference serving costs and architectural assumptions built around 32k–64k contexts are materially wrong.[1] The competitive frontier is shifting from model intelligence to the ability to serve 100k+ token contexts cheaply and fast at scale — a challenge that favors specialized hardware, tiered memory architectures, and new pricing models over raw parameter counts.[8] Decisions made now about infrastructure investment and pricing strategy will determine which players capture value from the agentic AI wave.

A wave of real-world usage data is forcing a reassessment of how expensive agentic AI actually is to serve. SemiAnalysis, analyzing 432k coding-agent requests from its own production environment, found the median input token count is 96k — not the 32k or 64k that most infrastructure planning has assumed.[1] More strikingly, roughly half of all requests in that dataset already exceed 128k tokens.[2] The culprit is not verbose users but the scaffolding agents build before a user types a single word: system prompts, tool and skill definitions, MCP schemas, and the rolling context of prior conversation turns and file contents that agents must carry to remain coherent across steps.[2] This structural inflation in prefill context length is quiet, automatic, and compounding.

The infrastructure consequence is a KV cache crisis. When a model processes a 96k-token request, the key-value tensors it generates are enormous — and GPU high-bandwidth memory, the fastest tier available, fills up quickly at scale.[3] The near-term engineering response is tiered memory: spilling KV cache from HBM into DRAM, NVMe, or network-attached storage, each tier slower and cheaper than the last.[4][5] NVIDIA has productized this approach through Dynamo, providing a naming taxonomy for the memory hierarchy.[3] Independent researchers have noted 1.4–1.7× wall-clock speedups at 98k context when attention computation is optimized, signaling that the attention mechanism itself — not just storage — is a primary cost driver at long context.[6]

The market is already repricing. GitHub Copilot announced it is retiring flat annual plans in favor of usage-based billing.[7] SemiAnalysis argues this is the leading edge of a broader shift: fast-tier inference products (Opus Fast, Gemini Flash) will proliferate, specialized inference silicon from Cerebras and Groq will see rising demand, and KV cache management will become a first-order competitive dimension for serving infrastructure.[8] The framing offered by multiple observers is that AI infrastructure is entering a "cloud economics" phase — where cost-per-unit matters as much as raw capability, and serving efficiency becomes the differentiator.[9] A complementary finding from the open-source community is that smaller models with well-designed guardrails can reach near-frontier accuracy on agentic tasks (an 8B model moving from 53% to 99% success rate), suggesting token-cost pressure may also push some workloads toward cheaper model tiers.[10]

The competitive implications extend beyond hardware vendors. SemiAnalysis has separately analyzed where value accrues as the inference bottleneck tightens, arguing that model labs — not cloud hyperscalers or inference resellers — are positioned to capture a disproportionate share of economic value from this transition.[11] Whether that holds depends partly on whether open-weight models like DeepSeek V4 (flagged as best-in-class for coding as of late April 2026) can compete on quality at the longer contexts that agentic workloads demand.[12] The thread connecting all of these dynamics is simple: the token counts are bigger than anyone planned for, and the industry is only beginning to adapt its hardware, pricing, and business models accordingly.