Google, Microsoft, OpenAI and Anthropic have announced support for a unified standard that lets web publishers opt out of having their content scraped for AI training. The standard aims to replace the separate instructions publishers currently need for each AI company’s web crawler, following calls from publishers and regulators for a simpler way to control how their content is used.
The standard will use a shared policy token in a website’s robots.txt file to communicate AI training preferences across participating providers. Microsoft is targeting February 2027 for implementation, Google is aiming for early 2027, and OpenAI plans to implement the standard in the coming months.