Four leading artificial intelligence companies have announced support for a new unified mechanism that would allow website publishers to control whether their content is used for AI training purposes. The initiative centers on a “policy token” signal proposed by Anthropic technical lead Kevin Kelley through the Internet Engineering Task Force. This signal would be integrated into websites’ existing robots.txt files, which have traditionally managed crawler access permissions for search indexing and archiving.
Currently, publishers seeking to prevent their content from being used in AI model training must establish separate preferences with each AI company individually. The new approach would consolidate these preferences into a single control that multiple AI providers can recognize and respect. OpenAI, Google, Microsoft, and Anthropic have stated they will honor the shared signal, with each company providing different implementation timelines. Anthropic has already adopted the standard, while Google and Microsoft indicate they will support it next year. OpenAI has committed to implementing the mechanism in the coming months.
The development reflects growing tensions between publishers and AI companies over unauthorized content scraping. Publishers have explored various defensive measures, including the Really Simple Licensing standard, which enables websites to charge AI crawlers for access, and services like Cloudflare that block such crawlers by default. Meta communications head Andy Stone also expressed support for the standardized approach, though the company did not confirm whether it would implement the signal.
Microsoft emphasized that its implementation will offer granular controls, allowing publishers to distinguish between AI training restrictions and restrictions on AI grounding—a process where models reference verified sources to improve accuracy. The company clarified that publisher opt-out preferences would not affect how websites appear in Bing search results. Google similarly noted that its previous controls enable sites to manage AI feature inclusion and training use while maintaining standard search visibility.
Article Attribution | Read More at Article Source
Article summary produced by Claude AI