Cloudflare to block mixed-use web crawlers by default from September 2026

Cloudflare just gave the AI industry a firm deadline: separate your web crawlers by September 2026. Starting on that date, the company will flip a switch on its default settings, automatically blocking any crawler that mixes search indexing, agent services, and AI training from accessing pages that carry ads. The announcement came Wednesday.

What that means in practice: unless a site owner deliberately changes their settings, these so-called mixed-use bots will be locked out by default. The new defaults apply to all new Cloudflare customers, any new sites set up by existing customers, and all existing free accounts.

The decision could rattle AI model providers who rely on scraping the open web for training data and powering agentic tools. Cloudflare points out that most site owners want to be found via search and even through AI services, but they don't want their intellectual property handed over for free.

Cloudflare takes a thinly veiled swipe at Google, the world's largest search engine, noting it has access to "about 2x more information" than other AI companies. The reason, Cloudflare argues: Google makes it hard for customers to stay discoverable without also being used for AI.

Google has pushed back on that characterization. It points to a bot called Google Extended that lets site owners opt out of having their content used for AI training or products like Gemini Apps and Vertex API, without affecting their Google Search rankings. Still, Google's flagship Googlebot does crawl for AI features like AI Overviews and AI Mode.

"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," said Cloudflare co-founder and CEO Matthew Prince, nodding to the recent milestone where bots outpaced humans online for the first time, a shift that wasn't expected until next year.

Prince added: "Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training."

Cloudflare has been rolling out a suite of tools to give publishers more say in the AI era. Last year it launched a marketplace where sites can charge AI bots for scraping, called Pay Per Crawl. That initiative is now evolving into Pay Per Use, which lets publishers charge AI companies when their content actually creates value, not just when it's fetched.

The change could also save bandwidth and compute resources. Cloudflare's data suggests that more than 50% of crawl traffic from AI crawlers is spent re-fetching pages that haven't changed.

To start, Cloudflare is working with two partners: Ceramic.ai and You.com. When a publisher opts in, they get paid when their content shows up in Ceramic's AI search results or when You.com accesses a piece of their premium content. Cloudflare says other AI companies can customize that model for their own needs.