OpenAI Says It Will Honor a Shared robots.txt Signal for AI Training Preferences, With Implementation Planned in the Coming Months

On October 30, 2026, OpenAI announced that it will honor a common signal in robots.txt, the file with which a website tells automated visitors what it prefers. The signal lets a site operator state a preference on the use of its content for foundational generative AI training. OpenAI calls the effort voluntary and says it makes it easier to give the same preference to several participating providers at once.

Until now, the post says, an operator had to identify each crawler and maintain separate rules for each provider. A shared signal reduces that work. OpenAI says it aims to implement the signal in the coming months and will publish another update with details of its implementation when it does, so today's post is a commitment and not a live feature. The new signal will not replace the controls OpenAI already provides through robots.txt. Its existing crawler controls will continue to apply and be respected.

The post links to an Internet-Draft that defines the signal, and the draft explains how it works. It is draft-kelley-aipref-training-preference-signal-00, dated 30 October 2026, an individual submission by K. Kelley, listed with an affiliation to Anthropic. The IETF page says the draft is not endorsed by the IETF and has no formal standing in the standards process. In it, an operator writes an ordinary robots.txt group whose User-Agent line carries a policy token, __usage_ai-training. In the draft's example, `Disallow: /` together with `Allow: /press-kit/` expresses a preference against AI training for every path except /press-kit/, where it expresses a preference in favor. The draft adds no new directive names or grammar, and a parser that does not know the token ignores the group, as it would ignore any group for an unknown user agent. A group written for User-Agent: * must not be read as matching the token, and a file without the token expresses no preference at all.

The same draft sets limits that matter. It says the token is not a directive about crawling: the crawler rules in robots.txt stay independent and keep their meaning, and a consumer of the signal may, but need not, use the preference to decide what to fetch. It says private agreements may override a preference expressed this way. It uses the working group's definition of AI training, which covers using content to modify the learned parameters of a generative AI model and leaves out models used only for non-generative tasks. OpenAI's post itself does not spell out the token, does not give a date, does not name the other providers it joins, and says nothing about content already collected. It states that OpenAI joins other participants in the IETF AI Preferences working group in the announcement, and that its practices will keep adapting as that work and the wider landscape evolve.

For site operators, the practical step is to wait for OpenAI's implementation details before relying on the signal, and to keep existing crawler rules in place, since OpenAI says they still apply. For readers, the news is a stated commitment to a shared signal, built on a draft that has no formal standing yet, and not proof that any content is protected today.