# Cloudflare Simplifies AI Bot Management with Bot Preference Sync Manage AI bot traffic on your website from a single dashboard and sync with robots.txt. - **Source:** [Cloudflare Blog](https://blog.cloudflare.com) - **Published:** 2026-08-21 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/cd0b67d9-3df8-4a8a-8713-c89a87e46d74 ## Challenges in Managing AI Bot Traffic Websites face the challenge of dealing with various AI bots serving different purposes. Some sites want to prevent their content from being used to train AI models, while others prefer search engine bots to index their content. Cloudflare introduces a new feature to simplify this complex management process. Bot Preference Sync allows users to manage AI bot policies from a single dashboard and automatically synchronize these policies with their robots.txt files. ## How Bot Preference Sync Works Cloudflare's new feature automatically reflects users' AI bot configuration settings in their robots.txt files. For example, if a user chooses to block training bots, this preference will appear as a 'Disallow' directive in robots.txt. The feature is available to all Cloudflare users, from free plans to enterprise customers. Users can turn the feature on or off at any time while preserving their existing robots.txt files. ## Flexibility for Different Business Models E-commerce sites may want their products to be recommended by AI assistants, while publishers relying on ad revenue might want to prevent their content from being used for AI training. Cloudflare enables users to set separate policies for search, agent, and training bots to accommodate these different needs. Bot Preference Sync ensures these policies are applied consistently across all layers of protection. ## Transparency and Verification Cloudflare demands transparency from AI bot operators. Bots that perform both search and training must provide additional information to remain unblocked even when 'Disallow Training' is set. This information includes URL-level visibility into which pages were used for training and metrics about search results. Cloudflare Radar publishes a list of bots that meet these transparency criteria. ## Challenges with Mixed-Use Bots Bots that combine search, agent use, and training under a single user agent make management difficult for website owners. Cloudflare states that non-transparent bots will continue to be blocked. Transparency is seen as a prerequisite for continued bot access. This approach increases control for site owners while encouraging bot operators to be more open about their practices. ## Compatibility with Existing robots.txt Files Rather than replacing existing robots.txt files entirely, Bot Preference Sync prepends users' AI bot preferences to their current files. This preserves any previously defined Disallow directives. By offering a dynamic solution instead of static file management, Cloudflare reduces user workload and ensures policy consistency. ## Options for Search and Agent Bots Users can choose from three options for search and agent bots: allow full access, block on ad-serving pages, or block everywhere. For training bots, the 'Disallow' option completely prevents content from being used to train AI models. Cloudflare ensures each of these options is correctly expressed in the robots.txt file. ## A Critical Step for the Modern Web The increasing use of AI bots creates new management challenges for websites. Cloudflare's Bot Preference Sync helps site owners address these challenges by automatically publishing settings made in a single dashboard, eliminating the risk of policy inconsistencies. This innovation provides significant convenience for businesses needing different AI bot policies. ## Sources covering this story - https://blog.cloudflare.com/bot-preference-sync/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._