Big Tech Adopts Shared Standard for AI Training Opt-Outs
Google, Microsoft, OpenAI, and Anthropic introduce a unified robots.txt signal for web scraping.
1Unified Opt-Out Mechanism
Google, Microsoft, OpenAI, and Anthropic have committed to honoring a single robots.txt line to manage AI training data. This shared signal replaces the current fragmented system where webmasters must write individual rules for every crawler. The initiative aims to streamline how publishers express preferences regarding their content usage for generative models.
2Scope and Limitations
The new standard governs content usage rather than preventing web crawling entirely. Bots can still fetch pages, but they must respect the policy token regarding training. Crucially, this mechanism does not address content already scraped, nor does it function as a formal, endorsed industry standard by the IETF at this stage.
3Implementation Timelines Vary
Anthropic has already begun implementing the signal, while OpenAI plans to follow in the coming months. Microsoft has set a firm date for February 1, 2027, and Google is targeting the same year. Meta remains supportive of the concept but has not yet committed to honoring the shared signal on its platforms.
4Technical Divergence Observed
Companies are integrating the token differently into their existing controls. Google plans to treat the opt-out as a rejection of Google-Extended, which could inadvertently impact search citations. Conversely, Microsoft is introducing a separate, page-level AI Grounding preference, allowing for more granular control over how content is used in its various products.
5Draft Origins and Status
The mechanism originates from an Internet-Draft authored by Kevin Kelley of Anthropic, utilizing the policy token __usage_ai-training. The IETF currently labels this as an individual draft without formal standing, scheduled to lapse on May 3, 2027. It is not a directive for crawling behavior, but rather a voluntary request for compliance.
6Legal and Ethical Context
This development follows intense legal scrutiny, including lawsuits from The New York Times against OpenAI and Microsoft. Internal documents from Microsoft have previously described AI scraping as a massive theft of labor. Despite these tensions, the industry is moving toward formalizing these requests to mitigate ongoing conflicts with content creators.
7Market Enforcement Trends
Beyond voluntary signals, the broader market is shifting toward stricter enforcement and monetization. Cloudflare has blocked AI crawlers by default for new customers since July 2025, enabling sites to charge for access. Meanwhile, European regulators are increasingly viewing robots.txt as a primary machine-readable tool for publishers to assert their digital rights.
8Future Outlook for Publishers
While this shared line represents progress for webmasters, it remains a request rather than a technical lock. It does not provide the financial compensation publishers have sought for years. Moving forward, site owners may still need to rely on login walls or IP rate limiting to fully restrict unwanted AI data collection.
Original article: Latest from TechRadar — Big Tech Adopts Shared Standard for AI Training Opt-Outs