A single machine-readable training preference sounds like one permission switch for the open web. It is not. The proposed __usage_ai-training token addresses a narrow question: whether participating services may use content to train generative models. It does not standardize crawling, indexing, retrieval or the way AI products cite sources. Google, Microsoft, OpenAI and Anthropic have different existing controls and rollout plans, and the distinction matters most when a publisher wants to limit model training without disappearing from AI-powered answers.

One shared signal has one narrow purpose

The October 2026 IETF Internet-Draft describes a policy token called __usage_ai-training inside an ordinary robots.txt user-agent group. A publisher can express a site-wide or path-specific preference with familiar Allow and Disallow rules. The draft adds no new robots.txt directive and does not change the established path-matching grammar.

The token is a preference about how content may be used, not a traffic-management instruction. Ordinary crawler-specific groups still govern fetching. A parser that does not recognize the policy token is expected to ignore that group, and a wildcard User-agent: * group does not express this particular preference. A robots.txt file without the token therefore makes no statement through this shared mechanism.

The proposal remains an Internet-Draft in the IETF AI Preferences work. It is not a finalized RFC, a universal technical enforcement system or a copyright licence. A provider must implement and honor the signal for it to affect that provider's use of content.

Sources: Machine-Readable Training Preference Signal: Internet-Draft · IETF Datatracker: draft status · RFC 9309: Robots Exclusion Protocol

Google: the training choice can affect Gemini grounding

Google says it intends to support the shared training signal in 2027. Its announcement says that when a publisher uses the shared token to opt out of AI training and does not explicitly specify Google-Extended, Google plans to interpret that choice as if Google-Extended had been disallowed.

That detail matters because Google-Extended is broader than a training-only switch. Google's crawler documentation says it governs whether content may be used to train future Gemini models and whether it may be used for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google also says Google-Extended does not affect a site's inclusion in Google Search or act as a Google Search ranking signal.

A publisher should therefore not assume that a shared training opt-out will leave Gemini grounding untouched. Google's existing Google-Extended control remains available, but it does not offer the same clearly separate training and grounding choices that Microsoft has announced.

Sources: Supporting a shared AI training control for websites · Google's common crawlers: Google-Extended · Machine-Readable Training Preference Signal: Internet-Draft

Microsoft: training and grounding are separate choices

Microsoft's October 30 announcement combines the shared AI Training preference with a separate AI Grounding preference. Training can be expressed in robots.txt for a site or path and can also be overridden on an individual page. Grounding is a page-level preference, not a site-wide robots.txt rule.

Microsoft describes three grounding choices: Allow, DisallowExtended and Disallow. DisallowExtended limits grounding to Microsoft-operated experiences and excludes certain grounding services Microsoft provides to third parties. Both training and grounding default to Allow when no applicable preference is published.

Microsoft says the new preferences will begin taking effect as it crawls from February 1, 2027. A March 1, 2027 transition deadline matters specifically to sites still relying on the older NoArchive or NoCache preferences; those publishers have additional page-level and Bing Webmaster Tools steps to preserve grounding restrictions. Microsoft says the new choices do not change eligibility for traditional Bing search results.

Sources: Your content, your choice: clearer preferences for training and grounding

OpenAI: SearchBot and GPTBot remain distinct

OpenAI says it plans to implement the shared training signal in the coming months and that its existing crawler controls will continue to apply. Its current documentation distinguishes OAI-SearchBot, which helps surface pages in ChatGPT Search, from GPTBot, which may collect content for model training.

OpenAI also documents ChatGPT-User separately for certain user-triggered page visits. These are not interchangeable uses. A publisher can currently allow OAI-SearchBot while disallowing GPTBot, preserving eligibility for ChatGPT Search while expressing a different preference about training. User-triggered retrieval is a separate pathway and should not be described as automatic crawling.

Sources: Making it easier for websites to express AI training preferences · Overview of OpenAI Crawlers

Anthropic: implementation has started, but rollout is still in progress

Anthropic's Kevin Kelley, the author of the Internet-Draft, said on the IETF AI Preferences mailing list that Anthropic had already started implementing support. He also said it would take time to add support across the company's services and work through edge cases. This is evidence of an implementation effort, not a claim that every Anthropic service already honors the token.

Anthropic's existing public crawler guidance distinguishes ClaudeBot for potential model-training collection, Claude-SearchBot for search, and Claude-User for user-initiated retrieval. Those existing controls complement the shared training preference rather than disappearing when the new signal is adopted.

Sources: [ai-control] New draft: draft-kelley-aipref-training-preference-signal-00 · Does Anthropic crawl data from the web, and how can site owners block the crawler?

A publisher's decision depends on the product, not just the token

A publisher that wants ordinary search discovery and AI answer visibility may make a different choice from one that wants to restrict model training. OpenAI currently documents separate search and training crawlers. Microsoft has announced separate training and grounding controls. Anthropic's existing crawler roles are also separated by purpose, although its shared-token rollout is still underway.

Google is the important exception to a simple one-switch model: Google-Extended covers both Gemini training and grounding. Its planned interpretation of the shared opt-out means a publisher must understand the Google-specific rule before assuming that training and grounding can be separated.

No setting guarantees that an article will appear in an AI answer, receive a citation or generate traffic. Eligibility is not a promise of inclusion, and a machine-readable preference does not create a licensing or payment agreement.

Sources: Making it easier for websites to express AI training preferences · Supporting a shared AI training control for websites · Google's common crawlers: Google-Extended · Your content, your choice: clearer preferences for training and grounding · Overview of OpenAI Crawlers · [ai-control] New draft: draft-kelley-aipref-training-preference-signal-00 · Does Anthropic crawl data from the web, and how can site owners block the crawler?

What website owners should do before editing robots.txt

Inventory the current robots.txt file, crawler-specific groups, page-level robots meta tags and HTTP headers. Write down the intended outcome for each provider: ordinary search crawling, AI search discovery, model training, user-triggered retrieval and grounding. Do not treat these as one permission.

Check the provider's own implementation documentation again before deploying a new policy token. OpenAI's timing is described as the coming months; Google says 2027; Microsoft names February 1, 2027; Anthropic says implementation has begun but will take time across services. These dates and commitments are not interchangeable.

If a site still uses Microsoft's NoArchive or NoCache preferences, it should review Microsoft's transition instructions before March 1, 2027. For all publishers, keep private or confidential material behind authentication and authorization. RFC 9309 is explicit that robots.txt is not an access-control mechanism.

For now, the prudent approach is to document the desired policy, retain current working crawler rules and test provider-specific consequences before changing a production file. A shared signal can simplify one part of the decision; it does not remove the need to make the rest deliberately.

Sources: Making it easier for websites to express AI training preferences · Supporting a shared AI training control for websites · Google's common crawlers: Google-Extended · Your content, your choice: clearer preferences for training and grounding · Overview of OpenAI Crawlers · Does Anthropic crawl data from the web, and how can site owners block the crawler? · RFC 9309: Robots Exclusion Protocol