OpenAI will honor a shared robots.txt signal for AI training preferences
The point
OpenAI says it will honor a common signal in robots.txt for websites to express preferences about use of their content for foundational generative AI training. One shared signal replaces naming individual crawlers and keeping separate rules for each provider.
Why it matters
Publishers get one way to state training choices across participating providers instead of per provider rules.
Key details
- Implementation is planned in the coming months, with details in a later update
- Existing OpenAI crawler controls in robots.txt will continue to apply
- The signal draws on the IETF AI Preferences working group
Original ~2 min → 25 sec here Source: OpenAI News, Oct 30, 2026