Cloudflare's New Policy Aims to Protect Publishers' Content

Cloudflare has introduced a new policy that will block mixed-use crawlers from accessing pages with ads, unless the site owner adjusts the settings. This move is expected to impact how AI model providers access web content for training purposes and could lead to a more sustainable ecosystem for publishers.

The new policy, which will come into effect on September 15, 2026, will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. Cloudflare points out that most website owners want their content to be discoverable via search and AI services, but they also want protections against having their intellectual property given away for free.

Impact on AI Model Providers

The move could significantly impact how AI model providers access web content for training purposes. Cloudflare specifically calls out the "world's largest search engine" (clearly a reference to Google) as having access to about "2x more information" than other AI companies because the search giant makes it difficult for customers to remain discoverable without being used for AI.

"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," said Cloudflare co-founder and CEO Matthew Prince.

Benefits for Publishers

The new policy is expected to benefit publishers in several ways. For example, it will give them more control over their content and allow them to charge AI companies for accessing their content. Cloudflare has also released a range of tools to help publishers combat AI bots, including a marketplace that lets websites charge AI bots for scraping, dubbed Pay Per Crawl.

• The Pay Per Crawl marketplace is evolving into "Pay Per Use," which will allow publishers to charge AI companies when their content creates value, not just when it's fetched.
• The change could also help conserve publishers' bandwidth and compute resources for AI model providers, as Cloudflare's data suggested that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages.
• Cloudflare is initially working with two partners, Ceramic.ai and You.com, to put this into action, and other AI companies can customize this model for how they work.