Four AI Firms Back Single Robots.txt Signal to Bar Training Use

OpenAI, Google, Microsoft and Anthropic have jointly proposed a single mechanism allowing media outlets and other websites to block the extraction of their content for training artificial intelligence models, according to the three supplied reports. The development sits squarely within AI copyright and regulation: publishers currently must file separate blocking instructions for each data-gathering robot, and the proposed alternative would let them enter one set of instructions recognised by any AI company adopting the protocol.

The Italian outlet AI4Business reports that the proposal is called the Machine-Readable Training Preference Signal, presented as an Internet-Draft to the IETF on 30 October 2026 by Kevin Kelley of Anthropic within the AI Preferences working group. It remains a technical draft and does not constitute a definitively approved IETF standard. The same source describes a common label, __usage_ai-training, identifying the purpose of use rather than a particular crawler, with example robots.txt lines of User-Agent: __usage_ai-training and Disallow: /, and an optional Allow line for exceptions such as a press folder. The Arabic-language reports from Hespress and An-Nahar do not mention the token name or these example lines. The signal expresses a preference on content use rather than an order to halt crawling, and AI4Business notes that a generic User-Agent: * rule does not suffice; the specific token must appear in the file.

Implementation will not be simultaneous across the four groups. Microsoft will begin applying it from 1 February 2027 and Google at some point in 2027, according to Hespress and An-Nahar. OpenAI said it will roll it out "in the coming months", while Anthropic said it has "already begun" deploying it. Microsoft said in a blog post that the setting applies only to future model training, does not otherwise prevent robots from crawling the internet, and that content extraction will be permitted until a publisher changes the setting. Meta spokesperson Andy Stone expressed support on X, writing that a shared signal conveying these preferences to many providers at once is an important step, and adding that the company looks forward to working with other AI companies on these standards and applying them across the sector.

The announcement comes amid growing criticism and legal battles from news publishers. In December 2023 The New York Times sued OpenAI and Microsoft, accusing them of using millions of articles without permission or payment to train ChatGPT. Cloudflare has automatically blocked AI software on new customers' sites since July 2025, and a programme called "Really Simple Licensing" allows publishers to charge robots that visit and copy their pages. The next milestones to watch are Microsoft's 1 February 2027 start date and Google's unspecified 2027 rollout.