OpenAI, Google, Microsoft, and Anthropic Back Unified AI Training Opt-Out in Robots.txt

OpenAI, Google, Microsoft, and Anthropic announced on October 30 that they will honor a single shared signal in robots.txt that lets website owners opt their content out of AI model training across all four companies' crawlers at once, The Verge reported. The policy token — __usage_ai‑training — is proposed as an IETF Internet‑Draft by Kevin Kelley, a technical lead for standards at Anthropic.

Today, a site that wants to block AI training crawlers must write separate rules for each company's bot: GPTBot for OpenAI, Google‑Extended for Google, Claude‑Web for Anthropic, and so on. The new standard collapses that into a single line: User‑Agent: __usage_ai‑training followed by Disallow: /, The Next Web explained.

Implementation timelines vary significantly. Anthropic says it has already started implementing support, Kelley wrote on the IETF mailing list. OpenAI says it aims to implement the signal "in the coming months." Microsoft targets February 1, 2027, and will replace the NoArchive and NoCache tags it has used since 2023. Google says 2027, The Next Web noted.

Google Search Advocate John Mueller wrote that if a site opts out using the token without mentioning Google‑Extended, Google will treat it as if the publisher had disallowed Google‑Extended directly, AI Weekly reported. Sites that allow Google‑Extended can still appear in Gemini app answers and receive search traffic.

The token is a preference signal, not a crawling directive. As the IETF draft states, "this policy token is not a directive for crawling behavior" — bots may still fetch pages even when training is disallowed, AI Weekly noted. As AI Weekly characterized it, OpenAI frames the mechanism as "a coordination mechanism rather than a security control" under RFC 9309, meaning non‑compliant scrapers can disregard it entirely.

There are other hard limits. Updating robots.txt does not remove material already collected into existing training datasets or trained models. The draft explicitly notes it is an individual submission, not endorsed by the IETF, and has no formal standing in the standards process. Private agreements between a site and an AI company can override it. Microsoft added that the new preferences "do not establish licensing or payment arrangements," per The Next Web.

Microsoft is layering on additional controls. Beyond the shared training opt‑out, the company is adding a separate page‑level setting for AI grounding — the retrieval step that pulls live web content into chat answers rather than training model weights. One option allows Microsoft's own products such as Copilot; it blocks the grounding services Microsoft sells to other companies, Microsoft wrote in its announcement.

Meta communications head Andy Stone voiced support for a common signal on X, The Verge reported, but did not say whether Meta would honor it. "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step," Stone said. Participants listed in the IETF draft include Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation, and Creative Commons.

For developers and site owners, the practical change is simplicity. Instead of maintaining per‑crawler rules for a growing list of AI bots, a single line expresses a training preference across the four companies that collectively dominate AI model development. Google noted the proposal is consistent with the EU's Code of Practice for general‑purpose AI under the AI Act, which recognizes robots.txt as a primary machine‑readable preference mechanism, according to The Next Web.

The token will also appear in the context of ongoing publisher demands for payment. In July, Cloudflare told AI crawlers to pay or get blocked. Its CEO has warned that bot traffic could reach 1,000 times human web traffic. The shared opt‑out standard doesn't solve the payment question, but it does give every site operator — from indie bloggers to major publishers — a single, clear mechanism to say no to AI training across the industry's largest players.