# Major Tech Firms Adopt Unified AI Training Opt-Out Standard A new robots.txt token simplifies how publishers restrict generative AI training data access. - **Source:** [The Next Web](https://brifing.si/source/afe24b20-b275-4c19-b9a6-8bb3c8babae7) - **Published:** 2026-10-31 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/ba394472-e723-4e31-91b3-7e40fd6c978b ## Unified Opt-Out Mechanism Google, Microsoft, OpenAI, and Anthropic have committed to supporting a shared AI training opt-out signal via robots.txt. This initiative, proposed by Anthropic’s Kevin Kelley, allows website owners to restrict generative AI training data usage through a single policy token, replacing the current requirement for managing individual crawler rules for every separate company. ## Technical Implementation Details The proposed standard introduces a new policy token, usage_ai-training, within the robots.txt file. Site administrators can define specific paths to exclude from training sets while keeping other areas accessible. This draft, currently an individual IETF submission, aims to provide a machine-readable preference that complements existing crawler controls rather than replacing them entirely. ## Microsoft’s Strategic Timeline Microsoft plans to honor these new preferences starting February 1, 2027, replacing their existing NoArchive and NoCache tags. The company is also introducing a granular page-level setting for AI grounding. This feature allows site owners to permit content usage for internal products like Copilot while blocking grounding services sold to third-party corporate customers. ## Google’s Phased Integration Google intends to support the signal by early 2027. If a site utilizes the new token without explicitly mentioning Google-Extended, the system will interpret it as a block on Google-Extended. Sites that permit Google-Extended will continue to appear in Gemini app responses, maintaining their visibility and potential for traffic through search engine results. ## Scope and Limitations The draft explicitly states that the token reflects content usage preferences rather than formal crawling orders. It does not establish licensing or payment arrangements between publishers and AI firms. Furthermore, Microsoft warns that blocking AI grounding services may significantly reduce content visibility in AI-generated experiences, though it does not impact standard Bing search rankings. ## Regulatory and Industry Context Google notes that this proposal aligns with the EU’s Code of Practice for general-purpose AI under the AI Act. The draft acknowledges contributions from major media organizations, including the Financial Times, BBC, and the European Publishers Council. Despite this, the IETF has not yet formally endorsed the submission, leaving its long-term standardization status currently uncertain. ## Broader Publisher Demands The push for standardized controls follows increasing pressure from publishers seeking compensation for their data. Companies like Cloudflare have previously challenged AI crawlers to pay for access, citing concerns over massive traffic spikes. Meanwhile, Australian broadcasters like ABC and SBS continue to advocate for direct payment models for news content used in AI models. ## Future Industry Implications While Meta has expressed support for a common signal, it remains unclear if they will adopt this specific standard. The move represents a shift toward balancing publisher autonomy with AI development needs. Moving forward, stakeholders must navigate complex edge cases and technical complications as these major tech companies work toward their respective 2027 implementation goals. ## Sources covering this story - https://thenextweb.com/news/ai-training-opt-out-robots-txt-google-openai-microsoft-anthropic — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._