Science & Tech

Major Tech Giants Unveil Unified Opt-Out Standard to Block AI Scraping

Ahmed Mohammed
Google took its next leap in artificial intelligence Wednesday with the launch of a project called Gemini that’s trained to think more like humans and behave in ways likely to intensify the debate about the technology’s potential promise and perils. (AP Photo/Jeff Chiu, File)
Google took its next leap in artificial intelligence Wednesday with the launch of a project called Gemini that’s trained to think more like humans and behave in ways likely to intensify the debate about the technology’s potential promise and perils. (AP Photo/Jeff Chiu, File)

At a Glance

  • OpenAI, Google, Microsoft, and Anthropic pledged on Friday to introduce a single, standardized protocol allowing media outlets and websites to block their content from being scraped for AI model training.

  • The new solution eliminates the need for publishers to implement multiple blocking commands for different web-crawling bots.

  • Rollout timelines vary among companies: Anthropic has already started, OpenAI plans rollout in the coming months, Microsoft sets implementation for February 1, 2027, and Google targets 2027.

  • While internet infrastructure provider Cloudflare has blocked AI bots by default on new customer sites since July 2025, tech giants noted that opting out of training may reduce a publisher's visibility in AI apps.

Major artificial intelligence companies are moving to address mounting pressure and legal battles from news publishers by introducing a centralized mechanism to manage content scraping.

Background and Legal Pressures

  • Surge in Web Crawlers: Publishers have faced an overwhelming rise in bots harvesting articles to train large language models, prompting legal action from prominent news organizations such as The New York Times, which sued OpenAI and Microsoft in December 2023 over copyright infringement.

  • Existing Alternatives: Beyond forthcoming protocol adoptions, internet infrastructure provider Cloudflare has automatically blocked AI bots on new customer sites since July 2025, and frameworks like Really Simple Licensing already permit publishers to charge visiting bots.

Scope and Limitations of the New Protocol

  • Default Settings and Training Only: Content scraping will remain permitted by default until publishers manually adjust the setting. Furthermore, the new control applies exclusively to future model training and does not completely stop bots from crawling web pages.

  • Visibility Trade-off: Both Microsoft and Google cautioned in separate blog posts that opting out of AI training could decrease the likelihood of a publisher's material surfacing within AI-driven applications and search experiences.

  • Industry Support: Meta spokesperson Andy Stone expressed broad industry backing on X, stating that creating a unified signal for multiple providers is a significant step forward.

FYI

The ongoing disputes largely hinge on the legal doctrine of fair use, with major tech firms arguing that training AI models on publicly available journalistic content constitutes transformative use, while publishers maintain that unauthorized usage undermines their revenue and intellectual property rights.

Dive Deeper: Handpicked Stories for You
Follow Channel8 for continuous updates:

Countries Navigate US-China Rivalry Over Artificial Intelligence

Ahmed Mohammed

Sun Nov 01 2026

Loading related news...