AnswersPublished24 sources
How the Shared robots.txt AI Training Opt-Out Works
On October 30, 2026, Google, OpenAI, Microsoft and Anthropic backed a shared robots.txt preference for AI training: User agent: usage ai training with Disallow: /. Microsoft says it will honor the signal from February 1, 2027; Google is aiming for early 2027, OpenAI gave no firm date beyond “the coming months,” and...
Published byEdited with GPT-6 LunaImages generated with GPT Image 2
Research answer
Key takeaways
- On October 30, 2026, Google, OpenAI, Microsoft and Anthropic backed a shared robots.txt preference for AI training: User agent: usage ai training with Disallow: /.
- Microsoft says it will honor the signal from February 1, 2027; Google is aiming for early 2027, OpenAI gave no firm date beyond “the coming months,” and Anthropic said it had begun implementation.[8][17]
- The signal is an IETF Internet Draft, not a completed standard. It addresses training preferences—not, by itself, whether pages can be crawled or appear in ordinary search results.[2][7]
AI Prompt
openai.comCreate a landscape editorial hero image for this Studio Global article: What did Google, OpenAI, Microsoft, and Anthropic commit to on October 30, 2026, regarding a shared robots.txt opt-out signal for generative. Article summary: On October 30, 2026, Google, OpenAI, Microsoft, and Anthropic backed a shared robots.txt signal for expressing whether website content may be used for generative AI training, rather than requiring publishers to address e. Topic tags: general, general web. Style: premium digital editorial illustration, source-backed research mood, clean composition, high detail, modern web publication hero. Use reference image context only for broad subject, composition, and topical grounding; do not copy the exact image. Avoid: logos, brand marks, copyrighted characters, real person likenesses, fake screenshots, UI text, readable text, watermarks, charts with fake numbers, clic
A shared robots.txt token could let publishers express an AI-training preference once instead of writing a separate training rule for every participating provider. On October 30, 2026, Google, OpenAI, Microsoft and Anthropic backed the proposal, which uses the token __usage_ai-training.4
8 The important caveat: it is a preference providers say they will honor, not a technical barrier that prevents access to a site.
The robots.txt rule and what it means
Anthropic standards lead Kevin Kelley submitted the Machine-Readable Training Preference Signal as an Internet-Draft to the IETF’s AI Preferences working group. It defines __usage_ai-training as a policy token that publishers can use in an ordinary robots.txt group.2
6
7
To express a site-wide preference against AI training, the example is:
User-agent: __usage_ai-training
Disallow: /
Publishers can also use ordinary path rules to express a more selective preference. For example, this disallows training use generally while allowing the /permitted/ path:
User-agent: __usage_ai-training
Disallow: /
Allow: /permitted/
The token names the use preference, not a particular company’s crawler. The draft requires the token to be named explicitly; a User-agent: * group does not match it.7
34
What the signal does—and does not—control
The shared token is specifically about whether content may be used for generative AI training. It is not, by itself, a request to stop crawling or remove a page from search results.2
7 Because it is a robots.txt preference, publishers should not treat it as a technical lock: its effect depends on participating providers honoring it.
7
8
That distinction does not mean every provider’s existing controls become irrelevant. Google says it plans to treat the shared opt-out as a Google-Extended disallow if a site has not specified a Google-Extended rule; its existing provider-specific control therefore remains important when setting Google preferences.33
34
Microsoft has separately said its new preferences will not change eligibility for Bing search results.19
31 That assurance is specific to Bing; the available sources do not establish a comprehensive search-indexing guarantee across all four companies.
When each company plans to honor it
The dates are announced plans, not evidence that every implementation is already live:
- Anthropic: Kelley said Anthropic had started implementing the signal. The available reporting does not establish a firm completion date.
8
- Google: Aiming to support it in early 2027.
- Microsoft/Bing: Plans to honor the shared training preference as it crawls pages starting February 1, 2027.
17
- OpenAI: Reported as planning support “in the coming months”; the supplied sources give no firm date.
Microsoft’s separate training and grounding settings
Microsoft is pairing the shared training preference with additional controls. Its Bing announcement describes AI grounding as a separate, page-level preference: publishers can manage whether content may help inform generated responses independently of training.17 The reported grounding choices are Allow, Disallow, and DisallowExtended. The last option is described as allowing grounding in Microsoft-operated experiences while excluding Microsoft’s third-party grounding services.
3
17
Microsoft’s transition dates matter for sites using its older NoArchive or NoCache tags. Reporting says temporary grounding preferences in Bing Webmaster Tools will be available to eligible sites from December 15, 2026, and identifies March 1, 2027 as the migration deadline for sites relying on those older tags.3
44
51 Microsoft’s new preferences are scheduled to take effect as Bing crawls content from February 1, 2027.
17
31
What remains unsettled
- The IETF process is unfinished. The proposal is an Internet-Draft, not evidence of a completed IETF standard. The cited draft is dated October 30, 2026, and is listed as expiring May 3, 2027.
2
34
41
- Past collection and training are not addressed in the supplied material. The available proposal and announcement summaries do not say whether the preference applies retroactively to content already collected, requires deletion, or changes models that have already been trained.
- The signal depends on provider compliance. The sources describe participating companies’ plans to honor a preference; they do not establish enforcement against nonparticipants.
7
- Meta is not established as a participant here. The supplied reporting names Google, OpenAI, Microsoft and Anthropic, but does not verify a Meta implementation commitment.
4
For publishers, the practical takeaway is to treat the shared token as a useful common training preference—not as a substitute for reviewing provider-specific rules, separate grounding settings, or each company’s rollout details.
Studio Global AI
Continue your research
This page includes a source-backed answer you can continue inside Studio Global.
People also ask
What is the short answer to "How the Shared robots.txt AI Training Opt-Out Works"?
On October 30, 2026, Google, OpenAI, Microsoft and Anthropic backed a shared robots.txt preference for AI training: User agent: usage ai training with Disallow: /.
What are the key points to validate first?
On October 30, 2026, Google, OpenAI, Microsoft and Anthropic backed a shared robots.txt preference for AI training: User agent: usage ai training with Disallow: /. Microsoft says it will honor the signal from February 1, 2027; Google is aiming for early 2027, OpenAI gave no firm date beyond “the coming months,” and Anthropic said it had begun implementation.[8][17]
What should I do next in practice?
The signal is an IETF Internet Draft, not a completed standard. It addresses training preferences—not, by itself, whether pages can be crawled or appear in ordinary search results.[2][7]
Sources
anythingengineoptimization.comAI-search news archive — AEO Wire