Cloudflare’s New AI Crawler Defaults Explained

Cloudflare’s September 15, 2026 policy change draws a sharper line between traditional search indexing and AI uses of web content. For newly onboarded domains, crawlers classified as Training or Agent are blocked by default on pages that display ads; crawlers classified as Search remain allowed. 215

That distinction matters for publishers that want to preserve discovery traffic while limiting automated systems that train models or retrieve content for an AI assistant in real time.

The default: block Training and Agent on ad-bearing pages

The new default applies to new domains onboarding to Cloudflare. On pages where Cloudflare detects ads, it blocks two categories of automated traffic:

• Training — crawlers that take content to train or fine-tune a model.
• Agent — automated activity acting in real time on someone’s behalf, such as chat-fetch bots and browser-use agents. 13

Search remains allowed by default. Cloudflare defines this category as crawling or indexing content so a service can answer questions about it later, a model it associates with expected referral traffic or other equitable compensation for the publisher. 13

This is therefore not an across-the-board block on AI. The default is limited to Training and Agent behavior, and it is limited to pages displaying ads. 215

Which Cloudflare customers are affected?

Cloudflare’s official documentation identifies the new default as applying to domains newly onboarding to its network. 215 Reporting on the rollout also describes the scope as including new sites added by existing customers and Free-tier sites that had not manually changed their settings. 23

Existing customers should not assume every established zone automatically receives the same policy. The practical outcome depends on the domain’s onboarding status, plan and existing configuration. Reviewing the zone’s AI crawler controls is the safest way to confirm the active policy. 23

The three traffic categories site owners can control

Cloudflare separates automated traffic by behavior rather than treating every AI-related crawler as one type. The three categories are independently configurable: 13

Category	What it covers	Default for new domains on ad-bearing pages
Search	Crawling or indexing content to answer questions later	Allowed
Training	Collecting content to train or fine-tune a model	Blocked
Agent	Real-time automation acting for a user	Blocked

For each category, a site can select one of three approaches:

1. Allow the traffic.
2. Block on pages that display ads.
3. Block on all pages. 23

That granularity lets a publisher, for example, keep search indexing available while blocking training crawlers across the full site—or apply the narrower advertising-page rule instead.

Why mixed-purpose crawlers are the key SEO risk

A crawler’s label is not necessarily limited to one behavior. Cloudflare says crawlers that combine Search and Training are evaluated according to all of their behaviors, with the most restrictive applicable rule determining access. 2615

This can have an important consequence: if a site blocks Training, a crawler that Cloudflare treats as both Search and Training may also lose its ordinary search-crawling access. Cloudflare’s documentation and rollout coverage identify Googlebot, Applebot and BingBot as examples of multipurpose crawlers relevant to this rule. 26

In other words, “allow Search” does not automatically preserve access for a bot whose activity is also classified as Training. A site owner considering a site-wide Training block should assess the potential search-indexing trade-off first. 24

What happens to the legacy “Block AI bots” control?

Cloudflare has said that configurations blocking AI training—including the legacy “Block AI bots” option—also apply to mixed Search-and-Training crawlers. 2 That makes a policy review especially important for sites that enabled the older broad control expecting it to affect only non-search AI activity.

Later coverage of the September rollout reported a newer “Disallow AI Training” approach intended to express a no-training preference while keeping Googlebot, Applebot and BingBot available for search when a crawler is designated “Accountable.” Because the behavior and migration path can depend on the selected setting, administrators should verify the current control shown in their Cloudflare dashboard rather than rely on an older toggle’s name. 4

Why Cloudflare separates search from AI consumption

The policy reflects a publisher concern: search engines traditionally crawl pages, index them and may send readers back to the source. Generative AI services can instead use content for model training or present retrieved answers in their own interface, with less direct referral value to the originating publisher.

Cloudflare frames Search as traffic for which publishers should expect referrals or equivalent compensation, while its Training and Agent categories describe uses that take or act on content in different ways. 13 Its controls are designed to let owners retain automation they consider valuable and limit automation they do not.

The central question is not simply whether a bot is “AI.” It is whether the crawler’s behavior supports content discovery and returns value to the site, or primarily consumes the site’s material for training or real-time AI tasks. 13

What site owners should do

Before changing a policy, check the consequences at the behavior level:

• Keep Search allowed if preserving indexing and potential referral traffic is a priority.
• Use Block on pages with ads when the goal is to protect monetized pages without applying a full-site restriction.
• Use Block on all pages only after considering whether a crawler has mixed Search and Training behavior.
• Revisit any legacy Block AI bots configuration, particularly if maintaining Google, Apple or Bing search visibility matters. 236

Cloudflare’s September defaults give publishers more control, but the classification of multipurpose crawlers means the configuration is also an SEO decision—not merely an AI-policy setting.