Cloudflare, a company that powers about 20% of all web pages on the web, has announced it is now blocking AI crawlers by default. Plus, it is offering a new model to allow AI services to pay content creators to crawl their content, named Pay Per Crawl.
Cloudflare wrote that they are the “first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default.” Now, new customers that sign up for Cloudflare by default will automatically block AI crawlers. Existing customers can block AI crawlers anytime with a single click in their Cloudflare dashboard. This shifts content scraping from an opt-out to opt-in format. There is a lot of buzz on Techmeme on this news.
This site uses Cloudflare extensively, so does 20% of the internet, so this is a big deal. Will I block AI crawlers, probably not – I don’t need to get paid for my content here. But most of you probably should (not all of you but many of you).
Pay Per Crawl – This is a new initiative from Cloudflare to let content creators set their rates and get paid by AI services that want to crawl your content. If you want to sign up for Pay Per Crawl, go here to learn more. Publishers can block all AI crawlers, allow specific ones, charge for access, or grant free access — full control over how their content is accessed. AI crawler owners will be able to use Pay Per Crawl to register, see pricing, and choose to pay or not. Pricing will set by both publishers, who can set rates, and AI companies, who can choose whether to access webpages at those rates.
Here are some of the publishers who are part of this announcement: ADWEEK, Atlas Obscura, BuzzFeed, Inc.,Fortune, Stack Overflow, News/Media Alliance, The Atlantic, Battelle
Media, Evolve Media, Hyperscience, IAB Tech Lab, O’Reilly Media, Quora, Raptive, Sovrn, Inc., StockTwits, Third Door Media, TIME, Webflow. Yes, Third Door Media is Search Engine Land and Martech (but not this site, I am not blocking anyone…)
Here are some of the settings in Cloudflare:
Here is an awesome video from Cloudflare’s CEO on why this is so important for the future of the web:
Forum discussion at X.
We all want to be satisfied, even though we know some people who will never be that way, and others who see satisfaction as a foreign emotion that they can’t hope to ever feel.
Newspaper Ads Canyon Crest CA
Click To See Full Page Ads
Click To See Half Page Ads
Click To See Quarter Page Ads
Click To See Business Card Size Ads
If you have questions before you order, give me a call @ 951-235-3518 or email @ canyoncrestnewspaper@gmail.com
Like us on Facebook Here
OpenAI ChatGPT Agent Marks A Turning Point For Businesses And SEO
OpenAI’s ChatGPT agent marks a change in how users interact with web pages and complete...
Daily Search Forum Recap: July 17, 2025
Barry Schwartz is the CEO of RustyBrick and a technologist, a New York Web service firm...
0 Comments