OpenAI, Google, Microsoft, and Anthropic announced they will adopt a new standard allowing website operators to communicate AI training preferences through a shared robots.txt signal. This shared "policy token" signal, proposed by Kevin Kelley, a technical lead for standards at Anthropic, will allow publishers to apply their AI training preferences across multiple AI companies simultaneously, rather than setting them individually for each provider. The signal will reside within a website’s robots.txt file, which has historically managed access for web crawlers.
OpenAI stated it will implement the standard in the "coming months," while Google and Microsoft plan to support it next year. Anthropic has already adopted the signal. The agreement means that the four companies will restrict their AI crawlers from accessing a site if they encounter this shared token.
Meta communications head Andy Stone signaled support for the new setting, stating, "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step." Microsoft detailed that its controls will offer granular options, enabling publishers to opt out of AI training or AI grounding, which is the process of directing an AI model to a verified source of information to reduce hallucinations. Microsoft also noted that websites will remain visible in Bing’s search results regardless of their stated preferences.
Google previously implemented similar controls that allow sites to manage whether their content appears in AI search features or is used for AI training, without affecting general search visibility.
