AI training opt-out: one shared robots.txt token for AI bots

A shared AI training opt-out for robots.txt got its big backers on October 30. Anthropic, OpenAI, Google and Microsoft say they will support one token, __usage_ai-training, instead of making site owners list every AI bot by name. It is one line to add. It is also a preference, not a lock, and most of the big four won't read it for months.

What actually happened

The token is defined in an IETF draft, "Machine-Readable Training Preference Signal," written by Kevin Kelley of Anthropic and discussed in the IETF's AI Preferences working group. You use it like any other robots.txt group:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That says: don't train on my site, except the press kit. The draft adds no new grammar, so existing parsers just skip it. It covers training of generative models only. It is not a crawl rule. The draft says private agreements can override it. It is also an individual draft at version -00, not an adopted standard.

Microsoft's Bing blog gives the clearest dates. Bing will honor the token from February 1, 2027. Microsoft also adds a separate page-level "AI grounding" setting (whether Copilot and Bing AI answers can use your page) and says neither setting changes your normal Bing search ranking. After March 1, 2027, Bing stops reading NoArchive and NoCache as AI signals.

SiliconANGLE reports that OpenAI will add support in the coming months and Google sometime next year. We could not confirm those dates on the companies' own pages, so treat them as estimates. Meta reportedly likes the idea but has not committed.

Why it matters for your business

For a store, this is a real choice and not a free one. Blocking training is easy to decide. Blocking grounding can take your products out of AI shopping answers. Those are now two separate settings, at least on Bing.

1. Decide per path, not per site. Product pages and docs may be worth sharing. Paid content and customer data are not.
2. Keep your per-bot rules. Until the token is live everywhere, GPTBot, ClaudeBot and Google-Extended entries still do the work.
3. Check your NoArchive tags before March 1. If you used them to keep content out of Copilot, they stop working.
4. Don't treat it as security. Fake AI crawlers ignore robots.txt. Rate limits and auth still matter.

Sources: IETF: Machine-Readable Training Preference Signal, Microsoft Bing blog, SiliconANGLE.