Major AI Firms Adopt Unified Opt-Out Standard for Web Scraping
Publishers gain a centralized mechanism to restrict AI training across multiple crawler platforms.
1Unified Opt-Out Mechanism
OpenAI, Google, Microsoft, and Anthropic have committed to a new standard for web scraping opt-outs. This initiative allows publishers to manage AI training preferences through a single shared policy token. By integrating this signal into the robots.txt file, companies aim to simplify the current fragmented process of individual crawler management.
2IETF Standardization Effort
The proposed policy token is spearheaded by Kevin Kelley, a technical lead for standards at Anthropic, within the Internet Engineering Task Force. This technical framework builds upon the existing robots.txt protocol, which historically governed search indexing and archiving. The goal is to create a universal signal that multiple AI providers can recognize.
3Addressing Publisher Challenges
Publishers currently struggle with the rapid influx of AI crawlers consuming their content for model training. Previous attempts to manage this traffic included the Really Simple Licensing standard and Cloudflare’s default blocking features. This new centralized approach provides a more standardized path for sites to protect their intellectual property from unauthorized scraping.
4Industry Adoption Timeline
Anthropic has already adopted the signal, while Google and Microsoft plan to integrate support throughout next year. OpenAI intends to implement the standard in the coming months. This phased rollout reflects the technical coordination required across major AI labs to ensure the shared token is honored consistently across their respective crawler infrastructures.
5Search Visibility Remains Intact
Microsoft clarified that opting out of AI training or grounding will not affect a website’s presence in Bing search results. Similarly, Google’s existing controls allow sites to restrict content usage for AI features without compromising search visibility. These measures ensure that publishers can protect training data without sacrificing their organic search traffic.
6Meta's Partial Stance
Meta communications head Andy Stone expressed support for the concept of a common signal for AI preferences. However, the company has not explicitly confirmed whether it will honor the new token. Stone noted that Meta looks forward to working with other industry players on the standards and their eventual implementation across the sector.
7Granular Control Capabilities
Microsoft is introducing granular controls that distinguish between AI training and AI grounding. Grounding involves pointing models to verified information sources to minimize hallucinations. By offering these specific options, Microsoft provides publishers with more nuanced authority over how their content is utilized by AI systems beyond simple binary access restrictions.
8Strategic Industry Shift
The transition to a shared policy token marks a significant shift in how AI companies interact with web publishers. By moving away from individual crawler management, the industry is attempting to reduce friction and legal tensions. This standard represents a critical step toward balancing AI model development with the rights of content creators.
Original article: The Verge — Major AI Firms Adopt Unified Opt-Out Standard for Web Scraping