Cloudflare's September 15 Deadline: How New AI Crawler Rules Reshape the Web

Starting September 15, 2026, Cloudflare will block any AI crawler that bundles search indexing with AI training or agent tasks under a single user-agent — unless the publisher explicitly opts in .

This isn't a minor settings change. Cloudflare, which manages about 20% of global web traffic, is flipping the internet's default from "opt-out" (crawlers allowed unless blocked) to "opt-in" for AI-specific crawlers . For publishers, it means enterprise-grade bot protection out of the box. For AI companies, it means the end of free, large-scale web scraping through Cloudflare-proxied sites.

The Core Change: September 15, 2026 Deadline

Cloudflare has set September 15, 2026 as the deadline for AI companies to separate their web crawlers into distinct, identified categories based on how the crawled content will be used. Any crawler that does not clearly declare its purpose — or that bundles search indexing with AI training under a single user-agent — will be blocked by default on Cloudflare-originated traffic, particularly on pages that display ads .

After this date, new Cloudflare customers and newly created websites will have default settings that allow only clearly identified search crawlers, while blocking AI training and AI agent crawlers unless the publisher explicitly opts in . Existing free-tier customers who do not manually update their preferences will also be migrated to the new defaults .

1. New Default Settings: From Opt-Out to Opt-In

Previously, the web operated on an opt-out norm: crawlers were allowed unless a website explicitly blocked them via robots.txt. Cloudflare has flipped this to an opt-in paradigm for AI-specific crawlers .

• For new customers and new sites: AI crawlers used for training or agentic actions are blocked by default. Publishers must actively enable them .
• For existing customers: The settings do not auto-change, but owners can (and are strongly encouraged to) migrate to the new three-category controls via the Cloudflare dashboard .
• On ad-supported pages specifically: Mixed-use crawlers (those that both index for search and scrape for AI training) will be blocked by default, because the default policy treats unlabeled or "mixed" traffic as training/agent traffic .

2. The Three-Category Crawler System: Search, Agent, Training

Cloudflare has introduced a new classification that lets publishers set independent rules for three bot behaviors, available to all customers including free-tier users :

Category	Definition	Default Setting (Sept 15)
Search	Crawlers that index content so it can appear in search results and send referral traffic back to the publisher.	Allowed by default — these bots drive readers and ad revenue.
Agent	Bots that act on behalf of a user (e.g., AI assistants performing tasks, shopping, or booking).	Blocked by default — publishers must opt in.
Training	Bots that scrape content solely to train AI models, without sending back users or attribution.	Blocked by default — publishers must opt in.

Publishers can now set a policy like "Allow Search, Block Training, Block Agent" from a single dashboard, even on the free plan . This is more granular than traditional robots.txt, which only allowed a blanket allow/deny per user-agent. Cloudflare's own documentation defines Search as activity where site owners "should expect referral traffic or other equitable compensation" .

3. Transition from Pay Per Crawl to Pay Per Use

Cloudflare first introduced Pay Per Crawl in July 2025 as a private beta — a marketplace where publishers could charge AI companies per request, using HTTP 402 ("Payment Required") status codes to negotiate access . This system allowed publishers to return a 402 response with pricing information when an unapproved crawler requested content .

By April 2026, during what Cloudflare called Agents Week (April 13-17), the company had evolved this into a broader Pay Per Use model, folding the per-crawl framework into a single operator console available across all plan tiers . The key distinction:

• Pay Per Crawl (2025 beta): A per-request microtransaction model — every crawl request triggered a payment negotiation via HTTP 402 responses .
• Pay Per Use (2026 GA): A more flexible system where publishers can set terms — flat licensing fees, per-crawl rates, or subscription-style access — and AI companies can manage access programmatically via Cloudflare's bot management and WAF rules .

The Stack Overflow implementation (February 2026) demonstrated the production version: their data licensing program uses Cloudflare's bot categorization to serve 402 responses to unauthorized crawlers, monetizing at scale without bespoke contracts .

How This Changes the Landscape

For Publishers

• New revenue stream: AI companies must now pay for training data from Cloudflare-proxied sites, rather than scraping for free .
• Granular control: Publishers can keep search crawlers (which drive ad traffic) while blocking training/agent bots — protecting both ad revenue and content value .
• Default protection: Smaller publishers who couldn't afford custom bot management now get enterprise-grade AI crawler blocking out of the box .

For AI Companies

• Cost of training data rises: The "free lunch" of web-scale scraping ends for sites on Cloudflare's network (~20% of the web) . AI companies must either pay publishers or negotiate licensing deals.
• Crawler identity crisis: AI companies must overhaul their crawler infrastructure by September 15 to separate search, training, and agent traffic into distinct, identifiable user-agents — or risk being blocked entirely .
• Google is most affected: Google's crawler has historically served dual purposes (search indexing and AI training for Gemini/SGE). Cloudflare's policy forces Google to either split its crawler or have its Googlebot blocked on ad-supported Cloudflare pages .

Broader Industry Impact

• Standard-setting move: Cloudflare is effectively defining a new protocol for AI crawler etiquette. Other CDNs and hosting providers may follow suit .
• Content Signals Policy: Cloudflare introduced a new set of machine-readable directives in robots.txt that go beyond page-level access, letting publishers signal whether their content can be used for search indexing, AI training, or AI agents . This is an extension of the traditional robots.txt to cover post-access usage, not just crawl permission.