Google, Microsoft, OpenAI and Anthropic have agreed to honour a single shared signal that lets a website keep its content out of generative AI model work. The agreement, reported on 31 October 2026, means site owners will no longer need a separate rule for each company. Until now, publishers had to block each firm’s crawler one at a time, using different names and different instructions. The new rule sits in a site’s robots.txt file, the small text file that tells automated visitors which parts of a site they may use.
The training signal comes from a draft proposal at the Internet Engineering Task Force, written by Kevin Kelley, a technical lead for standards at Anthropic. It uses a policy token called __usage_ai-training. A site adds a group for that token and then lists the paths it does not want used for model work, with Disallow: / covering a whole site. Allow lines can carve out exceptions, for example for a press kit. The draft frames the token as a statement of preference about how content is used, rather than an instruction about crawling itself.
Support arrives on different dates. Anthropic has begun work on the signal, and Kelley described it as a complement to the company’s existing crawler controls rather than a replacement for them. Microsoft will honour the preferences from 1 February 2027, replacing the NoArchive and NoCache tags it has used since 2023. Google aims to support the signal in 2027, with its crawler documentation pointing to early 2027. OpenAI said it aims to implement the signal in the coming months, and its existing controls, such as GPTBot, will still apply in the meantime.
How the training opt-out rule works
The rule is simple. A website owner adds a short block to robots.txt, naming the __usage_ai-training token and the paths to exclude. The four companies have pledged to read that block and to leave the listed content out of future training runs. That is a change from the current patchwork, where a publisher might block one company’s bot and miss another, or use a rule that one firm reads one way and another reads differently. One shared token removes that guesswork for the four firms that have signed up.
The rule has clear limits, and publishers who use it need to know them. It covers only future model work. It cannot undo content that has already been collected, and it does not stop normal crawling or remove a site from search results. A site that opts out can still appear in search, and its pages can still be visited by crawlers that fetch content for other purposes. The distinction matters because training and search are separate uses of the same pages, and the new token only speaks to the first of those uses.
Google has explained how it will apply the signal. It will treat the token as a stop sign for Google-Extended, the crawler that gathers material for its Gemini models. A site that opts out with the token, without mentioning Google-Extended, will be treated as blocking Google-Extended. Sites that allow Google-Extended can appear in Gemini app answers and get clicks, Google Search Advocate John Mueller wrote. Microsoft will offer separate controls for training and for grounding, when an AI tool looks up a verified source. It has also said websites will stay eligible for Bing regardless of their choice.
Why publishers wanted a shared training signal
Publishers have spent years fighting bots that scrape their articles. The New York Times has been suing OpenAI and Microsoft since December 2023 for using its articles without permission, and both companies invoke fair use as a defence. News sites and book publishers have argued that their work is being taken to build products that compete with them. The companies have argued that training on public material is lawful.
Cloudflare, which runs infrastructure for many websites, blocked AI bots by default on new customer sites from July 2025. A shared opt-out gives those owners a middle path. They can stay visible in search and in AI answers, if they choose, while still saying no to training. Microsoft and Google have warned that opting out could reduce the chance of a site’s content appearing in AI apps, so owners face a trade-off rather than a free choice.
What the training signal does not settle
The agreement does not end the legal fight over AI model work. Lawsuits over past use will continue, and the new token says nothing about payment for content that is used. It also says nothing about content that was already collected before a site opts out. Publishers who want money for past use, or who want old material deleted, will still need to pursue those claims through negotiation or the courts. The token is a forward-looking control.