OpenAI has announced it will honour a shared robots.txt signal that lets website owners express a single AI training preference to multiple AI providers at once — one of the most significant steps yet towards an industry-wide way of controlling how web content is used to train artificial intelligence models. Google matched the move the same day, saying it will adopt the shared AI training preference signal, while Microsoft rolled out new publisher controls this week and the proposal itself was authored by an Anthropic engineer. The fragmented world of AI opt-outs may finally be converging on one standard.
What OpenAI announced on 30 October
In its announcement, OpenAI described the effort as voluntary. Until now, a website operator that wanted to keep its content out of AI training had to identify each provider’s individual crawlers and maintain separate rules for every one. The shared AI training preference signal lets publishers communicate the same preference to multiple participating providers in one step.
OpenAI said it aims to implement the signal in the coming months. The company stressed that the new signal will not replace the robots.txt controls it already provides — its existing crawler rules will continue to apply and be respected — and encouraged other AI providers to honour the signal too.
How the shared AI training preference signal works
The signal is defined in an Internet-Draft titled “Machine-Readable Training Preference Signal” (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley of Anthropic, dated 30 October 2026 and set to expire on 3 May 2027. As an individual draft it carries no formal standing in the IETF standards process — but this is the concrete proposal that OpenAI, Google and Microsoft are rallying around.
The draft introduces no new directives and no new grammar. It works entirely inside the existing robots.txt standard (RFC 9309): a site owner binds a usage preference, such as “no AI training”, to website paths by writing ordinary robots.txt groups whose User-Agent line carries a special policy token, __usage_ai-training. The draft’s own example looks like this:
User-agent: __usage_ai-training Disallow: / Allow: /press-kit/
That block expresses a preference against AI training for the whole site except the press-kit section. Wildcard groups (User-agent: *) must not be interpreted as matching the policy token, and if a file contains no group with the token, no preference has been expressed at all. One distinction matters: the policy token is not a directive for crawling behaviour — content-usage preferences and crawler traffic-management signals are independent, so a page can in principle remain crawlable for search while carrying a training opt-out.
The token’s meaning comes from a companion draft, “A Vocabulary for Expressing AI Usage Preferences”, which defines AI training as using an asset to modify the learned parameters of a generative AI model. It sets out three categories — AI Training, AI Use, and Search — each with a preference of allowed, disallowed or unknown. There is an important caveat: the draft states plainly that it does not ensure preferences are followed and leaves the choice to the recipient. The AI training preference is, for now, a machine-readable request rather than an enforceable lock — which is why voluntary adoption by OpenAI, Google and Microsoft matters so much.
Google and Microsoft join the effort
Google’s announcement on the same day as OpenAI’s gave the proposal serious momentum. The company said it will support the shared AI training control along with other AI providers in 2027, and John Mueller of Google wrote that website owners will be able to use the shared policy token within robots.txt to set broad preferences across participating providers, as reported by Search Engine Land.
Google added a practical detail: when a website uses the policy token to opt out of training and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended, its existing AI training control. Other providers can handle it the same way.
Microsoft, meanwhile, announced new publisher controls through Bing — separate preferences for AI training and AI grounding, to be honoured as it crawls pages from 1 February 2027.
What the signal does not change
OpenAI’s existing crawler controls stay exactly as they are. GPTBot remains the crawler whose content may be used in training; OAI-SearchBot controls whether a site surfaces in ChatGPT’s search answers; OAI-AdsBot validates pages submitted as ads and does not feed training; and ChatGPT-User handles page visits triggered by user actions, where robots.txt rules may not apply. Each setting is independent, so a publisher can allow search visibility while disallowing training.
It is also worth remembering the limits of any robots.txt mechanism: it governs which addresses a compliant crawler may request, not what a system does with data it has already crawled. A shared AI training preference does not reach back to content already collected, and it only works for providers that choose to honour it.
What it means for Pakistani publishers and website owners
For Pakistan’s online media and tech sector, the change is practical. Until now, opting out of AI training meant maintaining separate robots.txt blocks for OpenAI’s GPTBot, Google’s Google-Extended, and every other provider’s crawler. A single shared AI training preference would collapse that into one block — a real simplification for newsrooms and small publishers without dedicated SEO teams.
The timing is notable. Pakistan’s technology industry is visibly accelerating: a Pakistani-rooted engineering firm, tkxel, was named a Select partner by both OpenAI and Anthropic this month, and OpenAI launched GPT-6.1 Terra, its half-price model for everyday computer work, just days ago. At the same time, a World Bank report found that 61 per cent of Pakistan’s service-sector workers hold jobs exposed to AI — so whose content trains these models is becoming a national economic question, not just a webmaster’s chore.
The next milestone is implementation: OpenAI says it will put the signal into effect in the coming months, and Google and Microsoft have committed to 2027. Publishers should not treat the draft as an immediate control — but the industry is clearly moving towards it.
Sources: OpenAI’s announcement of 30 October 2026, as reported by Unite.AI; Google’s publisher-control announcement via Search Engine Land.
Related Posts
TechnologyOneOffice and ZIEL Global Bring Affordable Cloud Office Suite to Pakistan
TechnologyPoco X8S Launches Globally: Compact 6.35-Inch Phone With Snapdragon 7 Gen 4, 100W Charging From $399
TechnologyEcoStar Launches 100-Inch QD Mini LED TV in Pakistan — Biggest Screen, Cinema-Scale 4K
TechnologyWorld Bank Proposes $50 Million Fund of Funds for Pakistan’s Startups
TechnologyEndeavor Pakistan Hosts Google’s Hashim Syed for AI Founders Series Session
TechnologyOGRA to Introduce AI-Based Monitoring for Pakistan’s Petroleum Supply ChainMost Popular
How-ToPTA Tax Calculator 2026: Check Mobile Tax on Any Phone335 views
SportsBallon d’Or 2026: London Ceremony, Nominees and All-Time Records Ahead of 70th Anniversary321 views
SportsUEFA Champions League 2026-27: Format, Teams and Schedule321 views
Canada JobsTech Jobs in Canada: Roles, Salaries, and Skills Needed311 views
Pakistan NewsPIA Boeing 787 Lease Deal: What It Means for the Airline297 views
ShowbizWomen’s Fashion in Pakistan 2026: Trends, Brands and Style Guide296 views
