AI-powered search engines (e.g., OpenAI’s ChatGPT, Perplexity) are failing to drive meaningful traffic to publishers while their web scraping activities increase. That’s one big takeaway from a recent report from TollBit, a platform that says it helps publishers monetize their content.
CTR comparison. Google’s average search click-through rate (CTR) was 8.63%, according to the report. However, the CTR for AI search engines was 0.74% and 0.33% CTR for AI chatbots. That means AI search sends 91% fewer referrals and chatbots send 96% less than traditional search.
Why we care. This is bad news for publishers because it shows AI search won’t replace traditional search traffic. As AI-generated answers replace direct website visits, you should expect to see this trend continue.
By the numbers. AI bot scraping doubled (+117%) between Q3 and Q4 2024. Also:
- The average number of scrapes from AI bots per website for Q4 was 2 million, with another 1.89 million done by hidden AI scrapers.
- 40% more AI bots ignored robots.txt in Q4 than in Q3.
- ChatGPT-User bot activity skyrocketed by 6,767.60%, making it the most aggressive scraper.
- Top AI bots by share of scraping activity:
- ChatGPT-User (15.6%)
- Bytespider (ByteDance/TikTok) (12.44%)
- Meta-ExternalAgent (11.34%)
- PerplexityBot continued sending referrals to sites that had explicitly blocked it, raising concerns about undisclosed scraping.
Context. One company, Chegg, is attempting to sue Google over AI Overviews. Chegg claims Google’s search feature has severely damaged its traffic and revenue.
About the data. There’s no methodology section, so it’s not entirely clear how many websites were analyzed, just that it’s based on “all onboarded ToolBit sites in Q4.” Toolbit says it “helps over 500 publisher sites.”
The report. TollBit State of the Bots – Q4 2024 (registration required)
Add Search Engine Land to your Google News feed. 
New on Search Engine Land
About the author
Danny Goodwin is Editorial Director of Search Engine Land & Search Marketing Expo – SMX. He joined Search Engine Land in 2022 as Senior Editor. In addition to reporting on the latest search marketing news, he manages Search Engine Land’s SME (Subject Matter Expert) program. He also helps program U.S. SMX events.
Goodwin has been editing and writing about the latest developments and trends in search and digital marketing since 2007. He previously was Executive Editor of Search Engine Journal (from 2017 to 2022), managing editor of Momentology (from 2014-2016) and editor of Search Engine Watch (from 2007 to 2014). He has spoken at many major search conferences and virtual events, and has been sourced for his expertise by a wide range of publications and podcasts.
We all want to be satisfied, even though we know some people who will never be that way, and others who see satisfaction as a foreign emotion that they can’t hope to ever feel.
Newspaper Ads Canyon Crest CA
Click To See Full Page Ads
Click To See Half Page Ads
Click To See Quarter Page Ads
Click To See Business Card Size Ads
If you have questions before you order, give me a call @ 951-235-3518 or email @ canyoncrestnewspaper@gmail.com
Like us on Facebook Here
OpenAI ChatGPT Agent Marks A Turning Point For Businesses And SEO
OpenAI’s ChatGPT agent marks a change in how users interact with web pages and complete...
Daily Search Forum Recap: July 17, 2025
Barry Schwartz is the CEO of RustyBrick and a technologist, a New York Web service firm...
0 Comments