Tech Policy Press - Cloudflare Wades into the Battle Over AI Consent and Compensation
CJL Director Courtney Radsch explains how Cloudflare's new policy to block AI crawlers by default and introduce a monetized marketplace for AI access marks a significant shift in web infrastructure, offering publishers more control and compensation but also raising concerns about the centralization of power and potential market dominance by Cloudflare.
Cloudflare, a company that handles about 20 percent of all internet traffic, took a significant step in the escalating war between humans and the AI companies that have treated the open web as a limitless, consequence-free training ground. It will now block AI crawlers by default—a quiet but radical reversal of the status quo where bots could crawl first and ask questions never. It also launched a Pay-Per-Crawl marketplace that lets a select group of publishers charge per‑page access fees to AI companies.
Last week’s announcements mark a deliberate turn away from the 'scrape‑now‑ask‑later' model that has underpinned large swaths of generative AI development, embedding consent and compensation directly into web infrastructure. In one move Cloudflare gave publishers back something they’ve been desperately trying to reclaim: control, consent, and compensation.
Cloudflare’s default setting is especially notable: new domains automatically block AI bots unless explicit permission is granted. Existing domains can toggle the setting on but will not have their default settings changed. This effectively makes opt-in the default for crawling, something publishers around the world have sought as copyright exceptions for text-and-data mining for AI turn copyright on its head by requiring opt-out rather than opt-in as the default for AI training. Furthermore, Cloudflare is introducing cryptographic bot verification and transparency dashboards—collectively allowing publishers to see who is crawling their site, how often, and whether it returns referral traffic.
Spoiler alert: referral traffic has plummeted dramatically, and few think that it is ever coming back as generative AI summaries and chatbots become the new interfaces for searching for information and interacting online. The collapse of referral traffic and audience reach will impact not just digital advertising revenue but also conversion rates. I’ve heard from countless publishers and collective management organizations (which represent rights holders) that referral traffic is down significantly, while bot traffic is up.
Cloudflare’s data paints a stark picture of how AI crawlers operate at indefensibly extractive levels: OpenAI’s scraping-to-referral ratio is 1,700:1 and Anthropic is at 73,000:1. Traditional search engines—particularly the market-dominant Google—till drive traffic, but increasingly, initial answers come from AI models that summarize content without redirecting traffic. Google’s crawler now scrapes around 14 pages per referral, down from 6 to 1 six months ago (and 2 to 1 ten years ago) even as its total search impressions increased nearly 50 percent a year after it launched Google AI Overview (and as it was judged to be liable for having an illegal monopoly in search).
These figures crystallize what publishers and creatives have long argued: AI development is built atop uncompensated creative labor and poses an existential threat to those who provide the content that makes the Internet such a valuable public resource.
Bot traffic not only fails to generate revenue but also imposes costs on the target website (think hosting, bandwidth and overage fees), strains their servers, degrades site performance and user experience (think slower load times or broken pages and even sites), and pose security risks. It can even knock entire sites or archives offline. One major sports site reportedly got 13 million AI-bot visits per month but received just 600 human users—a ratio that underscores the unsustainability of current AI crawling.
“The change in traffic patterns has been rapid, and something needed to change. This is just the beginning of a new model for the internet,” Stephanie Cohen, Cloudflare’s Chief Strategy Officer, said in an interview with Reuters.