Cloudflare and OpenAI Launch Joint Research Pilot to Make AI Search Smarter with Real-Time Network Signals

Cloudflare "strongly encourages" companies that do all three to run separate, identifiable crawlers for each purpose .

Upcoming default settings (September 15, 2026)

Starting September 15, 2026, Cloudflare will enforce new default settings for all new customers, new sites from existing customers, and free-tier accounts [8, 9, 10]:

• Training crawlers → Blocked by default on ad-monetized pages.
• Agent crawlers → Blocked by default on ad-monetized pages.
• Mixed-use crawlers (bots that combine search, training, and/or agent functions in one pass) → Blocked by default on any page that hosts ads [8, 10].
• Search-only crawlers → Allowed by default, so sites remain discoverable in traditional search.

The sharpest change targets mixed-use crawlers: these bots collect content for search and AI training in a single pass, making it impossible for site owners to tell which purpose the data serves. Cloudflare's default block forces AI companies to either split their crawlers or lose access to ad-supported pages [8, 10].

Pay Per Use: experiments with publisher compensation

On July 1, 2026, Cloudflare also announced it is shifting from its earlier Pay Per Crawl model (announced in 2025) toward a Pay Per Use model [10, 20]:

• Pay Per Crawl (private beta in 2025): charged AI companies each time a crawler requested content, using HTTP status codes and authentication .
• Pay Per Use (announced July 2026): publishers get paid when their content actually powers an AI answer, not just when it is crawled . This is described as a "metered utility" for AI search — the AI company pays for the value it delivers, not the raw access [19, 20].

The Pay Per Use model is still described as an "early experiment" and a "research program" . Its exact pricing, revenue splits, and rollout timeline for general availability have not been published.

How the pieces fit together

The OpenAI research pilot, the three-category taxonomy, the default blocks on mixed-use crawlers, and the Pay Per Use experiments are all part of one coherent strategy [8, 9, 20]:

1. Force transparency – by requiring separate crawlers for search vs. training vs. agents, site owners can finally see what each bot is doing.
2. Give site owners control – defaults block the most extractive behaviors (training, agents, mixed-use) while preserving discoverability (search allowed).
3. Create a compensation layer – Pay Per Use replaces the broken "crawl for free, get referrals" bargain with payment tied to actual AI answer generation.
4. Make AI search smarter – the OpenAI pilot uses Cloudflare's real-time signals to reduce wasteful crawling and improve the timeliness and accuracy of AI answers, which benefits both publishers (better attribution) and AI companies (lower infrastructure cost).