Cloudflare Draws a Line on AI Crawlers: Category Defaults and Publisher Marketplaces

Cloudflare has restructured AI crawler traffic into three distinct categories—Search, Agent, and Training—and set a September 15, 2026 deadline after which new domains on its platform will block Agent and Training bots by default on ad-supported pages [1]. Mixed-use crawlers that combine Search and Training functions can be blocked in full if a site owner opts out of Training access [1]. Alongside these defaults, Cloudflare has launched a pay-per-crawl marketplace letting publishers set minimum prices for AI crawler access [3][2]. More than half of all Cloudflare web requests now originate from AI agents, giving the company both the incentive and the leverage to restructure this market [3].

Publishers who previously had no practical tool to distinguish or charge for different types of AI scraping now have a default-block mechanism and a monetization path, both intermediated by Cloudflare. This places Cloudflare in a structurally powerful position as the policy and payment layer between AI companies and the web's content base, a role no single actor has held before.

Cloudflare has drawn a formal boundary across AI crawler behavior that the industry had previously treated as a single undifferentiated activity. Under the new framework, crawlers are classified as Search (helps people find content), Agent (helps AI systems complete tasks on behalf of users), or Training (absorbs content into models). Starting September 15, 2026, new domains on Cloudflare will have Agent and Training bots blocked by default on ad-supported pages, while Search crawlers remain allowed [1]. If an AI company operates a mixed-use crawler that serves both Search and Training functions, blocking Training can result in the crawler being blocked entirely [1]. The policy essentially forces AI companies to choose: separate their infrastructure by purpose or lose access.

Simultaneously, Cloudflare has built a pay-per-crawl marketplace that lets content owners set minimum prices for AI crawler access, positioning the company as a neutral intermediary rather than simply an enforcer [2][3]. The scale of the problem gives this some urgency: Cloudflare reports that more than half of its web requests now originate from AI agents [3], a figure that frames the policy not as a niche publisher complaint but as a fundamental shift in what the web's traffic actually is. Reuters and Nieman Lab both covered the marketplace launch in July 2025, treating it as a new economic layer for publisher-AI relations [4][5].

The policy has surfaced two distinct tensions. The first is structural: publishers who block Agent bots to protect their content may simultaneously reduce their visibility in AI-generated answers and discovery tools, creating a tradeoff that smaller sites in particular may find difficult to navigate [6]. The second is compliance: the classification system only works if AI companies actually separate their crawlers by purpose, and there is no clear enforcement mechanism beyond Cloudflare's bot-identification infrastructure. How AI companies respond to the September deadline—by splitting crawlers, negotiating marketplace terms, or seeking workarounds—remains the live question.

Observers broadly read the move as overdue. The Neuron framed it as converting a blurry permission into a structured system: search indexing, agent task completion, and model training are different deals that happened to be bundled into one unasked-for arrangement [1]. Digiday covered the tension between Cloudflare's self-positioning as a neutral marketplace facilitator and its simultaneous role as a unilateral policymaker setting defaults for millions of sites [7].