Cloudflare Introduces Granular AI Training Controls for Websites
New settings allow site owners to block AI training while maintaining search visibility
1The search vs. AI dilemma
Website owners previously faced an impossible choice between allowing AI training crawlers to access their content or risking reduced search visibility. This occurred because major tech companies used mixed-use crawlers that served both search indexing and AI training purposes. Cloudflare's new solution addresses this by separating these functions while maintaining search discoverability.
2Accountable crawler program
Cloudflare has established criteria for 'Accountable' AI crawlers that must provide: opt-out mechanisms for both training and summaries, URL-level transparency about content usage, and assurance that opting out won't affect search rankings. Apple, Google, and Microsoft have already qualified, while Amazon, Anthropic, Meta, and OpenAI are working toward compliance with these standards for their separate training crawlers.
3New control settings
Cloudflare now offers four settings: Allow (all crawlers permitted), Disallow AI Training (blocks training while allowing search), Block on pages with ads (conditional blocking), and Block (complete restriction). The Disallow AI Training setting specifically targets mixed-use crawlers from Apple, Google, and Microsoft while maintaining their search functionality.
4Implementation timeline
Starting September 15, Cloudflare will migrate existing settings automatically. Domains that previously blocked AI bots will transition to Disallow AI Training, preserving search functionality. New domains will receive preset configurations based on whether they rely on ad revenue, with ad-supported sites getting more restrictive defaults.
5Platform-specific implementations
Applebot uses 'Applebot-Extended' in robots.txt for training opt-outs. Google provides a webmaster toggle for generative search exclusions. Microsoft currently relies on NOARCHIVE meta tags but plans full robots.txt support by early 2027. All three companies confirm that training opt-outs won't affect search rankings.
6The advertising dimension
Ad-supported sites face unique challenges since AI answers may replace page visits where ads are viewed. Cloudflare's 'Block on pages with ads' setting specifically addresses this by detecting ad-serving pages. However, there's no equivalent Disallow AI Training option for ad pages due to technical limitations in robots.txt implementation.
7Impact measurement
Cloudflare Radar will track compliance and provide reporting on how accountable crawlers respect site owner preferences. This transparency layer helps publishers verify whether their opt-out choices are being honored in practice, addressing a key limitation of traditional robots.txt implementations.
8Agent crawler distinction
User-directed agents (like chat fetch bots) represent a separate category from search/training crawlers. Cloudflare currently doesn't offer Disallow settings for agents due to lack of established standards, but will revisit this as protocols like ai-prefs mature.
9The summary challenge
While training controls address content usage, AI summaries present separate distribution challenges. Early data shows summaries can reduce overall site visits while increasing conversion rates for those who do visit. Cloudflare plans to develop more granular controls for summary appearances beyond simple opt-outs.
10Publisher vs. retailer impacts
Different business models are affected differently by AI summaries. Advertising-reliant publishers may prioritize visit volume, while e-commerce sites might prefer higher-intent visitors. Cloudflare aims to provide visibility tools to help each business type make informed decisions about summary inclusion.
11Industry collaboration
Cloudflare emphasizes the need for infrastructure providers, content creators, and standards bodies like IETF to collaborate on open protocols. The current solution represents an interim step while broader industry standards for AI content usage continue to develop.
12Future roadmap
Next priorities include developing more sophisticated controls for AI summaries, allowing site owners to specify how much content appears in summaries rather than binary opt-outs. Cloudflare also plans to enhance reporting tools to help businesses understand how summaries affect their traffic patterns and conversion metrics.
Original article: Cloudflare Blog — Cloudflare Introduces Granular AI Training Controls for Websites