Cloudflare, a prominent provider of internet infrastructure, has announced a significant change to its approach regarding AI web crawlers. The company will now block access by known AI crawlers by default, aiming to protect web content from being accessed without permission or compensation, as detailed in a recent announcement.
Under the new policy, Cloudflare will inquire with new domain owners about their preferences regarding AI scrapers. Furthermore, the company is introducing a “Pay Per Crawl” model that allows publishers to set fees for AI crawlers seeking access to their content. AI users will have the option to review these prices and choose to register for the program or forgo the opportunity entirely. Initially, this program will be accessible only to a select group of leading publishers and content creators, with promises from Cloudflare to facilitate the use of quality content in a manner that ensures appropriate permissions and compensation.
Cloudflare has long been proactive in assisting domain owners in managing AI crawlers. The company allowed websites to block such crawlers starting in 2023, specifically targeting those that respect the robots.txt file—an agreement that, while standard, lacks enforceability. In a further enhancement, Cloudflare has enabled websites to block all AI bots indiscriminately, regardless of their adherence to the robots.txt protocol, a feature newly instituted as a default setting for new users. This functionality identifies scrapers through a comparison with a list of known AI bots. Additionally, Cloudflare has implemented a measure that directs web-crawling bots into an “AI Labyrinth,” designed to hinder unauthorized scraping activities.
Several high-profile publishers, including The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora, have embraced Cloudflare’s latest restrictions on AI crawlers. As reliance on AI chatbots for information grows, many websites are adapting to this shift in user behavior. “Trust in AI has increased over the past six months, resulting in a decline in engagement with original content,” stated Cloudflare CEO Matthew Prince during a recent Axios Live event.
Moreover, Cloudflare is collaborating with AI firms to assist in verifying their crawlers, enabling these companies to transparently communicate their purposes for content usage—be it for training, inference, or search functions. This information allows website owners to assess and make informed decisions about which crawlers they permit.
“Protecting original content is vital, as it is a cornerstone of the Internet’s value,” Prince remarked in the press release. “AI crawlers have been accessing content without limits. Our initiative aims to empower creators while fostering innovative growth in AI.”
