On October 30, 2026, major AI providers announced or began work on a shared way for websites to express preferences about using content to train generative AI models. The proposal is useful, but it is not a universal switch: it remains an Internet-Draft, provider rollout dates differ, and a training preference is not an access-control system. This guide separates the standards work from the implementation plans and explains why website owners should review search and grounding consequences before changing production rules.

What actually changed on October 30?

Four major providers have moved the proposal beyond a single-company discussion, but they are not at the same stage. OpenAI says it plans to implement the signal in the coming months; Google says it intends to support it in 2027; Microsoft says its new preferences will take effect as it crawls from February 1, 2027; and Anthropic's Kevin Kelley says the company has begun implementing support while warning that rollout across services will take time.

The proposed benefit is coordination: a publisher could communicate one broad training preference to services that choose to implement the same convention. The important qualifier is participation. A standards draft, a company commitment and a deployed crawler implementation are three different stages, and none should be presented as universal adoption.

Sources: Making it easier for websites to express AI training preferences · Supporting a shared AI training control for websites · Your content, your choice: clearer preferences for training and grounding · [ai-control] New draft: draft-kelley-aipref-training-preference-signal-00 · Does Anthropic crawl data from the web, and how can site owners block the crawler?

How the proposed policy token works

The draft describes an ordinary robots.txt user-agent group that identifies a purpose rather than a particular web crawler. A policy token named __usage_ai-training expresses a preference about training-related uses of site content; familiar allow and disallow path rules indicate where the preference applies. The proposal does not add a new directive or change the core robots.txt path-matching rules.

This is different from blocking a network request. A participating provider interprets the group as guidance on using information for training, while ordinary crawler-specific groups can still control how that provider fetches pages. If the draft changes during standards discussion, published examples and implementation instructions may change too.

Sources: Machine-Readable Training Preference Signal: Internet-Draft · IETF Datatracker: draft status

Is this an official IETF standard?

No. On October 31, the IETF Datatracker identifies the proposal as an active individual Internet-Draft and explicitly states that it has no formal standing or IETF endorsement. Its publication date is October 30, 2026. A document hosted by the IETF is not automatically an adopted RFC.

That distinction has practical consequences. Technical definitions and implementation behavior should be rechecked against official provider guidance before a site changes production settings. Website owners can study the draft now without treating the current text as a final compliance checklist.

Sources: Machine-Readable Training Preference Signal: Internet-Draft · IETF Datatracker: draft status

Search visibility and AI training are different decisions

A publisher may want readers and search engines to discover an article while expressing a separate preference about training AI systems on it. OpenAI documents distinct functions for OAI-SearchBot, GPTBot and user-requested ChatGPT-User visits. Google distinguishes ordinary Googlebot crawling from Google-Extended, a product token associated with certain AI uses.

Google says its existing Google-Extended rules do not determine inclusion in Google Search or affect its ranking. However, Google-Extended also governs certain Gemini training and grounding uses, so a shared training opt-out may have a product-visibility consequence unless the provider-specific rule is explicitly considered. Microsoft has announced a different design with separate training and grounding preferences. The details are not interchangeable across providers.

A useful decision starts by listing which products you want to remain discoverable in, which uses you want to restrict, and which provider controls currently implement those choices. A companion article in this series compares the Google, Microsoft, OpenAI and Anthropic approaches.

Sources: Making it easier for websites to express AI training preferences · Supporting a shared AI training control for websites · Overview of OpenAI Crawlers · Google common crawlers and Google-Extended · Your content, your choice: clearer preferences for training and grounding · Does Anthropic crawl data from the web, and how can site owners block the crawler?

What robots.txt does not protect

Robots.txt is a public instruction file, not a password, firewall, legal license or digital-rights enforcement service. RFC 9309 states that the Robots Exclusion Protocol is not a method of access authorization. If a confidential document is available without authentication, naming its path in robots.txt does not make it private.

The shared training preference also cannot guarantee what every scraper will do. A service must decide to read and honor the token. Copyright, contractual permissions and other legal questions are separate from the technical signaling mechanism. For sensitive files, the right solution remains authentication, authorization and storage access controls.

Sources: IETF Datatracker: draft status · RFC 9309: Robots Exclusion Protocol

A practical checklist for a small publisher

First, review the existing robots.txt file and identify active crawler-specific groups. Save a dated copy of the current configuration before making changes. Second, decide separately whether your goal is search discovery, AI answer visibility, model-training restrictions or protection of private material. These goals should not be collapsed into one policy.

Third, follow the current provider documentation for controls that are actually supported today. Fourth, monitor the shared-token draft and provider implementation notices; only deploy the new token when its behavior matches your intent. Fifth, test the effects on crawling and discovery rather than assuming a policy entry will be interpreted identically by all services.

If you operate a CMS or multiple domains, keep a record of which sites publish which rules and who is responsible for reviewing changes. That maintenance record is more reliable than relying on a one-time edit that may become outdated.

Sources: Making it easier for websites to express AI training preferences · Supporting a shared AI training control for websites · Machine-Readable Training Preference Signal: Internet-Draft · Overview of OpenAI Crawlers · Google common crawlers and Google-Extended

Why this matters beyond a technical file

The debate is about who gets to decide how publicly accessible information may be reused. Independent publishers need a clear way to state their choices; AI companies need consistent technical definitions; readers still benefit from discovering useful writing. None of those interests is fully resolved by adding a new robots.txt group.

A shared signal could reduce operational friction if enough services implement it. For now, its value is a credible direction of work, not universal protection. The sensible near-term response is to understand your existing controls, avoid publishing confidential material openly, and follow the actual rollout rather than the headline.

Sources: Making it easier for websites to express AI training preferences · Supporting a shared AI training control for websites · Machine-Readable Training Preference Signal: Internet-Draft