😸 Cloudflare draws an AI bot line
The Neuron · Grant Harvey · 2026-07-06
Cloudflare introduces granular AI traffic controls separating bots into Search, Agent, and Training categories, with new defaults blocking Agent and Training bots on ad-supported pages for domains joining after September 15, 2026.
Extraction
Topics: ai-crawlersweb-content-policybot-managementcloudflare
Claims
- Cloudflare introduced new AI traffic controls separating crawlers into three distinct use categories: Search, Agent, and Training.
- Starting September 15, 2026, new domains on Cloudflare will block Agent and Training bots by default on ad-supported pages while keeping Search bots allowed.
- Multi-purpose crawlers that combine Search and Training functions can be blocked entirely if a site owner blocks Training.
- AI companies have been treating the open web as undifferentiated content to crawl without distinguishing between search indexing, agent use, and model training.
Key quotes
AI companies have treated the open web like a pantry. Site owners are starting to label the shelves.
This turns 'AI crawling' from one blurry category into a permissions system. Search helps people find you. Agent bots may help a person complete a task. Training bots absorb your content into a model. Those are different deals.