Big AI firms back one robots.txt opt-out for AI training
Four of the world’s biggest AI companies have agreed on a single way for websites to say no to AI training. OpenAI, Google, Microsoft and Anthropic said on Friday 30 October that they will honour one shared signal in robots.txt, the file that tells automated visitors what they may do on a site.
The signal is a policy token called __usage_ai-training. A website that wants out adds two lines to its robots.txt file — User-Agent: __usage_ai-training, then Disallow: / — and every participating company is meant to read that as a refusal of AI training. The old company-by-company rules, such as Google-Extended for Google and GPTBot for OpenAI, keep working alongside it, and sites can still make exceptions for particular pages.
The idea comes from an Internet-Draft published on 30 October by Kevin Kelley, a standards lead at Anthropic, through the Internet Engineering Task Force. It is still a draft, not a finished standard, and it is voluntary: it only binds the companies that choose to follow it, and it does nothing about content already collected or already used to train a model.
Why it matters
Until now, publishers had to block each company’s crawler by name. One shared token cuts that chore to a single line. But the fine print matters: the token covers training only. It does not stop a site being crawled, appearing in search results, or being quoted in AI-written answers. Google says a site that uses the token without naming Google-Extended will be treated as if it blocked Google-Extended. Microsoft is going further, adding separate controls for AI training and for AI grounding, the step where a model is pointed at a trusted source to answer a question.
What happens next
Nothing changes yet. Anthropic, whose employee wrote the draft, has started adding support, according to reports. OpenAI says it will follow in the coming months, Microsoft will start honouring the signal on 1 February 2027, and Google aims to support it in 2027. Microsoft is also replacing its old NoArchive and NoCache tags, and sites using them have until 1 March 2027 to switch. Anthropic has not published its own announcement, so its backing is reported rather than confirmed in its words. Meta, another crawler, has welcomed the idea but has not committed to honouring the token.
This article is for general information and education only. It is not legal or financial advice.
Sources
- https://www.skimwire.com/ai-training-opt-out-robots-txt-openai-google-microsoft-anthropic/ (Skimwire, 30 October 2026, updated 31 October 2026)
- https://rrtechpulse.com/stories/robots-txt-ai-training-opt-out-signal/ (RR Tech Pulse, 30 October 2026, updated 31 October 2026)
- https://techprolonged.com/2026/10/openai-google-microsoft-and-anthropic-back-one-ai-training-opt-out/ (TechProlonged, 30 October 2026)
- https://aiweekly.co/alerts/openai-google-microsoft-anthropic-back-one-ai-opt-out-token (AI Weekly, 30 October 2026)
