Google, OpenAI, Anthropic and Microsoft plan to honor a common signal in robots.txt so their AI does not train on a website
IA en un minuto newsroom · Editor: Jon Elgezabal
In 30 seconds
Anthropic, OpenAI, Google and Microsoft are planning to accept the same robots.txt line that lets a site opt its content out of AI training. Kevin Kelley, of Anthropic, proposed it to the IETF; Anthropic has already started applying it, Google expects to join in 2027 and Microsoft, on February 1, 2027. Meta has not said if it will sign up.
A single line in robots.txt (the text file that tells search engines where they should not go) would let a website ask that AI not be trained on its content, if the companies do what they have announced. Kevin Kelley, of Anthropic, submitted a draft on October 30 to the IETF, the body that writes the internet's technical standards, for its working group on AI preferences. According to the trade press collected by Martin Cid Magazine, Google, OpenAI and Microsoft plan to honor it.
Until now, a site owner had to write a different rule for each company, using names such as GPTBot, ClaudeBot or Google-Extended. The draft does not invent new syntax: in an ordinary group of rules, the User-Agent line carries the identifier __usage_ai-training instead of the name of a crawler. A "Disallow: /" states that no training is wanted on anything on the site, and an "Allow" line opens exceptions, such as a press kit. It has two limits: the wildcard rule "User-agent: *" does not count for this identifier, so a robots.txt that blocks every bot today expresses no preference about training, and the signal does not regulate crawling, only how content is used.
The timelines differ. Anthropic has begun to implement the identifier, though Kelley wrote that rolling it out to every service it runs will take a while. Microsoft will honor it only for crawls carried out after February 1, 2027; from March 1 it will no longer treat its NoArchive and NoCache tags as training or grounding signals. Grounding is a separate preference, configured page by page, and Microsoft warns that refusing it will noticeably lower a site's visibility in its AI answers. Google expects to support the identifier in 2027 and, for a site that uses it without mentioning Google-Extended, will act as though Google-Extended had been blocked. OpenAI has promised to follow in the coming months. Meta has not said whether it will honor it; its head of communications, Andy Stone, called a common signal "an important step" in comments to The Verge, as reported by Martin Cid Magazine.
According to Google and Microsoft, the signal does not prevent a site from appearing in their search results. It also does not undo what has already been crawled, and the draft still has to go through the IETF working group before becoming a standard, so it does not require any company to honor it.
Why it matters · analysis and opinion
For anyone running a website, what matters is that the request no longer means learning the name of each crawler. But it remains a request: no company is obliged and the draft still has to go through the IETF. The practical catch is the wildcard, because a current robots.txt that blocks every bot says nothing about this signal; whoever wants to opt out will have to add the new group and first check what rules already exist for Google-Extended, since Google applies the signal only if the site does not mention it. The difference between training and grounding also counts: declining the second can cost visibility in Microsoft's AI answers. Meta's absence is a reminder that the reach will depend on who joins, and that what has already been crawled falls outside the signal.
Source: Martin Cid Magazine · Written with the help of AI: how we make the news
