AI and web content: a new signal to decline model training

OpenAI, Google, Microsoft and Anthropic are involved in an effort to make it easier for website owners to express their preference against using content to train generative AI. The proposal arrives as US newspapers sue Brave over alleged use of their articles. Both developments raise a fundamental question: who may use information available online, and on what terms?

A common preference for AI training

On October 30, 2026, Kevin Kelley of Anthropic published an individual Internet-Draft with the Internet Engineering Task Force (IETF), which contributes to the development of internet standards.

The proposal aims to make it simpler for website publishers to express opposition to the use of their content for generative AI training. Site owners can already set rules for particular crawlers and providers, but managing separate mechanisms is complicated.

The draft introduces a common purpose-specific signal in robots.txt, the file traditionally used to communicate instructions to crawlers. A site could declare:

User-agent: __usage_ai-training
Disallow: /

These lines express a preference against using content from the whole site to train generative AI models. Publishers could allow particular paths while declining others. One signal could communicate the same preference to multiple participating providers.

Source: IETF technical proposal

A preference signal, not a technical barrier

The mechanism does not technically prevent a crawler from downloading a site's content. It expresses a preference about its use for training. Allowing access to a web page does not necessarily mean allowing its use to train a model.

Companies that commit to honoring the signal need to account for it in their processes, but it does not block actors who choose to ignore it. It does not by itself create a new legal protection or a payment system for content.

The October 30 document is an individual Internet-Draft, not an adopted standard. Adoption and implementation depend on each company. OpenAI says it aims to implement the signal in the coming months; Google aims to support it in 2027. Microsoft says it will honor the new preferences from February 1, 2027 as it crawls pages. Anthropic's Kevin Kelley authored the proposal; no implementation date is established here.

OpenAI announcement · Google announcement · Microsoft announcement

Training an AI and using content in an answer are different operations

The proposal mainly addresses model training. AI systems can also use web information in another way: when answering a question, an assistant may retrieve documents to supplement its response.

This approach is used in architectures known as Retrieval-Augmented Generation, or RAG. Unlike training, which changes a model's learned parameters, RAG supplies additional information when the model generates an answer. A system can consult a recent article without that article having been part of the model's original training.

These practices raise different questions about permission, publishers' rights and possible compensation. Microsoft has also announced a separate AI Grounding preference to let publishers control the use of pages in generated responses from its AI services. This additional Microsoft control is separate from the proposed common training signal.

Four US newspapers sue Brave

On November 2, 2026, two lawsuits were filed in the United States against Brave Software. One was brought by The New York Times; the other by the New York Daily News, the Chicago Tribune and the Orlando Sentinel.

Brave is known for its browser, but it also operates Brave Search, a search engine with its own index, and a commercial API through which applications and AI systems can access search results.

The publishers allege that Brave copied their articles without permission to build its index and sold access to AI companies. The complaints mention Anthropic, Perplexity, Mistral, xAI and Cohere as companies using certain Brave services; those companies are not the defendants in these new lawsuits.

The allegations also concern purported reproduction of content in answers generated by Brave's tools and a browser feature that may bypass some subscriber-only access controls. These are allegations made by the publishers, not facts established by a court decision.

Source: TechCrunch, November 2, 2026

Brave's distinctive crawler approach

According to its official documentation, Brave Search does not use a differentiated crawler user agent that makes it easy for sites to identify it. Brave says this avoids discrimination by sites that allow only Google to crawl them. It also says its bot will not crawl a domain or page that Googlebot cannot crawl.

The approach illustrates a wider tension: publishers want control over different uses of their content, while search services need access to information to build indexes. The question becomes more complex when those indexes help supply AI-generated answers.

Source: Brave Search crawler documentation

Meta also under scrutiny in the United Kingdom

On October 30, 2026, the UK Competition and Markets Authority (CMA) opened an investigation into proposed commercial terms for businesses using Meta's platforms and services.

According to the CMA, the terms could allow Meta to collect content from websites of businesses and news publishers using its services and use it to develop its products and services, including AI features. The authority is examining whether the terms could unfairly disadvantage businesses that depend on Meta's services to reach audiences.

Meta disputes that interpretation and says the changes primarily concern its advertising and commercial services. The rollout has been paused pending consultation. No breach of competition law has been established at this stage.

Source: UK Competition and Markets Authority, October 30, 2026

What this means for developers using AI APIs

For developers, SaaS companies and AI integrators, these developments go beyond crawlers. They highlight operations that need to be distinguished:

• Crawling: retrieving pages or documents from the internet.
• Indexing: organizing content so it can be searched.
• Training: using data to change a model's parameters.
• RAG: supplying documents to a model to supplement an answer.
• Redistribution: making content or reproductions available to third parties, potentially through a commercial service.

Each operation has different technical, contractual and legal implications. A search API can supply information to an application without necessarily granting permission to redistribute it in full. Likewise, using a pretrained model through an API does not automatically mean that documents sent in a request will train that model.

For projects using multiple AI providers, source traceability and understanding each provider's terms are increasingly important. Technical accessibility alone does not settle which uses are permitted or who bears responsibility.

Toward more precise governance of content in the AI era

A common signal could reduce the technical burden on publishers and make it easier to apply consistent policies across providers. But it addresses only part of the issue.

A preference mechanism does not guarantee that everyone will respect it, establish the applicable legal terms or determine how content might be compensated. The Brave lawsuits and the Meta investigation show that tensions now extend across the entire chain, from collection to use in commercial products.

For organizations building or integrating AI services, the challenge is to combine access to quality information with respect for the rights and choices of those who create it. Progress means making publishers' preferences understandable, actionable and verifiable across an international ecosystem of providers and services.