Cloudflare’s Disallow AI Training Lets Publishers Keep Search While Restricting AI Training

Cloudflare’s September 15, 2026 launch of Disallow AI Training changed the practical choice facing publishers: they can seek to remain discoverable in search while telling a qualifying mixed-use crawler not to use their content for model training. The companion Accountable designation identifies operators that either already honor that preference or have made a concrete commitment to do so. 5

The central change: purpose, not just bot identity

Cloudflare’s controls divide crawler activity into three independently managed purposes:

• Search: crawling to index content for search results.
• Training: taking content to train or fine-tune an AI model.
• Agent: retrieving content while carrying out a user-requested action. 10

That distinction matters because one crawler infrastructure can support more than one purpose. Before this approach, blocking a crawler associated with AI could also jeopardize ordinary search discovery. With Disallow AI Training, a site owner can make a narrower choice: allow search access while refusing training use. 5

What “Accountable” means

Cloudflare created the Accountable designation for bot operators that can separate search indexing from training use in a way that respects a site owner’s preference. Apple, Google, and Microsoft either honor the new setting or have committed to honor it on Cloudflare’s stated timetable. 5

For publishers, the designation is significant because it is intended to prevent an otherwise useful search crawler from being treated as fully blocked solely because its operator also has AI-training uses. In other words, qualifying mixed-use operators can continue to index pages for search after the publisher disallows training. 5

Cloudflare’s available announcement describes the standard at a high level, but does not provide enough detail here to verify every operator’s implementation date, individual crawler behavior, or full technical commitments. Publishers should therefore validate their own bot settings and crawl data rather than assume identical behavior across all operators.

Which traffic can remain—and what is blocked

The new setting is designed to preserve search indexing for Accountable mixed-use operators while rejecting training use of the same content. Cloudflare explicitly frames the feature as a way to remain indexed for search while refusing training by that crawler. 5

The supplied material does not establish a complete, crawler-by-crawler public allow/block list. It does support these practical conclusions:

• Search access from qualifying Accountable operators can remain available when training is disallowed. 5
• Training and Agent categories can be blocked separately from Search. 10
• A general “block all AI bots” setting remains a broader option for site owners who prefer a pause or a blanket restriction, but it is less precise than purpose-based controls. 7

That makes the September update especially relevant to sites that depend on organic visibility: it reduces the need to choose between a training opt-out and search presence.

Advertising-supported pages and the September defaults

Cloudflare had previously announced that, beginning September 15, 2026, new domains onboarding to Cloudflare would have Training and Agent crawlers blocked by default on pages that display ads. 10 The policy reflected a publisher-focused premise: ad-monetized content may warrant more restrictive defaults for AI uses.

The earlier approach also raised an important risk. When a crawler had multiple purposes, a restrictive category-level block could affect search crawling too. The Disallow AI Training and Accountable model is Cloudflare’s answer to that conflict: publishers can protect ad-supported content from training while retaining search indexing where the crawler operator can honor the distinction. 5

The available sources do not fully document how every pre-existing domain was migrated or which defaults applied to every account type. Owners of established domains should review their own configured policies rather than infer them from the new-domain default.

Google-Extended, Applebot-Extended, and Bing’s workaround

Purpose-specific controls address a problem that publisher-specific directives have tried to solve: separating an operator’s search activity from its AI use.

Cloudflare has discussed Google’s proprietary opt-out mechanisms, including Google-Extended and nosnippet, while noting publisher feedback that those mechanisms had not prevented all uses publishers wanted to control. 22 The material provided for this report does not independently establish the precise scope or current behavior of Google-Extended, Applebot-Extended, or Bing’s NOARCHIVE directive.

The broader takeaway is clearer than the implementation details: a dedicated opt-out or an Accountable commitment can support the instruction, “index this page for search, but do not use it for training.” A blunt crawler block cannot reliably deliver that distinction.

Why finer controls matter now

Cloudflare’s own traffic analysis classified 80% of AI crawling over the prior 12 months as training, compared with 18% for search and 2% for user actions. The company says the classification is based on operator disclosures and industry sources. 16

That distribution helps explain why publishers may want separate settings instead of a single AI on/off switch. A publisher might accept search indexing, reject permanent incorporation of content into a model through training, and make a separate decision about real-time agent access.

Cloudflare also reported in 2025 that more than 2.5 million websites had chosen to completely disallow AI training through its managed robots.txt feature or managed AI-crawler blocking rule. 13 The September controls offer those publishers a more granular path where preserving search visibility is also a priority.

What comes next: control over AI summaries

Cloudflare has said it aims to give publishers more granular control over AI-generated summaries in early 2027. 2 The supplied announcement confirms the direction, but not the final product design, enforcement method, or exact publisher options.

For now, the operational lesson is straightforward: review Search, Training, and Agent policies as separate decisions. If organic discovery is important, do not assume that an AI restriction must also restrict search—especially when a crawler operator is designated Accountable. 5