The one-crawler problem: how Google's search bot became an AI pipeline and why publishers are fighting back

The CMA's response: a world-first conduct requirement

In October 2025, the UK's Competition and Markets Authority (CMA) designated Google as having Strategic Market Status (SMS) in general search and search advertising, given its roughly 90 percent share of the UK search market . On 3 June 2026, the CMA imposed a Publisher Conduct Requirement (CR) — the first binding obligation under the UK's Digital Markets, Competition and Consumers Act 2024 .

Key elements of the CR:

• AI opt-out for publishers. For the first time, publishers can prevent their content from powering AI features in Google Search (AI Overviews, AI Mode) without giving up search visibility .
• Fine-tuning opt-out. Following consultation feedback, Google must also allow publishers to opt out of having their content used for AI model fine-tuning .
• Mandatory attribution. Google must clearly link back to publisher content in AI-generated search results .
• Transparency and compliance. Google must publish compliance reports every six months for the first year, with key data and metrics, and must proactively notify publishers when it extends controls to new AI services .
• Implementation deadline. Google has nine months (by March 2027) to implement all changes, though the CMA expects key controls sooner .

The CMA explicitly considered but rejected crawler separation — forcing Google to use different crawlers for search indexing versus AI ingestion — as disproportionate. It concluded the cost to Google would be materially higher and that improved controls were an equally effective and more proportionate remedy .

Cloudflare's position: crawler separation is the only real fix

Cloudflare, which handles traffic for about a fifth of the web, submitted a detailed response to the CMA consultation arguing the CR is a step forward but does not fix the structural problem . Its core arguments:

• The CR delegates control back to Google. Publishers remain dependent on Google's proprietary systems and cannot use third-party blocking tools (like Cloudflare's own Crawl Control) without risking search rankings .
• Crawler separation is the only technical solution. Google should be required to use a separate crawler for AI training and grounding, so publishers can block one without harming the other .
• Granular controls aren't enough. The CMA proposal distinguishes between "AI training/grounding" and "search generative AI features" in a technically confusing way that most publishers will not understand .
• Size of the problem. Googlebot accounts for roughly 50 percent of all web crawls, and Google's AI search scrapes 3.2 times more content than OpenAI does .

Cloudflare CEO Matthew Prince personally traveled to London in October 2025 to lobby the CMA for this approach and has been publicly pushing the "unbundling" of search and AI crawlers since then . Cloudflare also announced that from 15 September 2026, it will block mixed-purpose crawlers on ad-supported pages for relevant customers .

Where things stand now

• The CMA imposed the CR on 3 June 2026, giving publishers an AI opt-out tool for the first time . The substantive control obligations come into force on 3 December 2026 and the requirement remains in force until 10 October 2030 .
• Cloudflare continues to argue that the CR is insufficient and that only mandatory crawler separation will restore true publisher autonomy .
• Google has said it is testing new controls that let publishers manage AI use, and warned that "breaking Search" by forcing crawler separation could harm the user experience . Google's own response to the CMA consultation argues that the CR strikes the right balance .
• The CMA has left the door open to further action, saying it is actively monitoring how Google implements these changes and may take additional steps if needed .

The battle over Google's crawler is not just a UK story. The CMA's decision is being watched closely by regulators in the EU, the US, and other jurisdictions, and the outcome could set a global precedent for how search dominance and AI training intersect.