OpenAI said on October 30 that it will honor a shared robots.txt signal for foundational generative AI training preferences. It aims to implement support in the coming months, rather than announcing a completed rollout.
What changes
The voluntary approach is intended to let website operators communicate one preference to participating providers instead of maintaining a separate rule for each. OpenAI says its existing crawler controls will remain in place.
A draft, not a finished standard
The linked training-preference proposal is an individual Internet-Draft dated October 30. The IETF record explicitly says it is not endorsed by the IETF and has no formal standing in its standards process. It describes purpose-specific policy groups using existing robots.txt syntax.
MintedBrain analysis
For a small publisher, a shared preference could reduce maintenance work. That benefit depends on adoption and implementation. Keep a record of current crawler rules and check provider documentation before changing them; this announcement alone does not establish universal support.
For related building work, see our full-stack app prompt pack.
Sources: OpenAI announcement, October 30, current OpenAI crawler documentation, and the Internet-Draft linked above. No crawler configuration was changed or tested for this report.
Discussion
Sign in to comment. Your account must be at least 1 day old.