Gary Illyes said some search engines can be confused by seeing double slashes in a URL. Previously, Google suggested not to use double slashes in your URLs but here Gary is saying that technically pages load with double slashes but with some search engines may have issues crawling URLs with...
The boom of generative AI products over the past few months has prompted many websites to take countermeasures. The basic concern goes like this: AI products depend on consuming large volumes of content to train their language models (the so-called large language models, or LLMs for...
Google has updated its Verifying Googlebot and other Google crawlers help document to add a new section describing the three categories or types of crawlers they have. They have their Googlebot crawler, special-case crawlers and user-triggered crawlers.I believe this was done after we,...
Google's John Mueller was asked if Google Search would be impacted if you block other crawlers, like Bing or the ChatGPT crawler. John said no, "Google doesn't care if you block other crawlers."This came up on Twitter the other day, where the SEO asked, "if I block CCBot from crawling my...
