Google recently updated the documentation of its Google-Extended web crawler user agent, reflecting changes in product naming and clarifying the impact on search, which may be a concern for those who choose to block the crawler. The updated documentation offers clearer guidance on controlling content access for use in AI model training.
Google-Extended User Agent
Introduced on September 28, 2023, Google-Extended offers web publishers a user agent that can be used to control how their sites are crawled. Publishers can allow or disallow the Google-Extended user agent using the Robots Exclusion Protocol, giving them a way to opt-out of having their content scraped and included in AI training datasets.
Google describes Google-Extended as a “standalone product token” but that’s non-standard terminology for how publishers understand the concept of User Agents.
The original announcement described the new user agent:
“Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products.
By using Google-Extended to control access to content on a site, a website administrator can choose whether to help these AI models become more accurate and capable over time.”
Blocking Google-Extended is done with the “Google-Extended” User Agent:
User-agent: Google-Extended
Disallow: /
Google Changelog
Google keeps a changelog of important updates made to guidance and communication with web publishers and the search marketing community. The changelog of Google’s developer pages announced a change to the Google-Extended documentation.
The revision comes after the renaming of Bard to Gemini Apps, specifying that Google-Extended’s indexing now contributes to Gemini Apps and Vertex AI generative APIs. The new wording reassures publishers that this does not affect Google Search, addressing potential concerns about the possible implications from opting out of Google-Extended AI data collection.
What Changed?
Google’s changelog clarifies that Google-Extended crawling is exclusive to Gemini Apps and has no impact on Google Search.
The Changelog advises:
“Updated the description of the Google-Extended product token
What: With the name change of Bard to Gemini Apps, we clarified that Gemini Apps is affected by Google-Extended, and, based on publisher feedback, we specified that Google-Extended doesn’t affect Google Search.”
The updated guidance no longer uses the Bard brand name, switching it out to Gemini. And the following sentence was added:
“Google-Extended does not impact a site’s inclusion or ranking in Google Search.”
Read Google’s updated crawler overview:
Overview of Google crawlers and fetchers (user agents)
Featured Image by Shutterstock/Ribkhan
We all want to be satisfied, even though we know some people who will never be that way, and others who see satisfaction as a foreign emotion that they can’t hope to ever feel.
Newspaper Ads Canyon Crest CA
Click To See Full Page Ads
Click To See Half Page Ads
Click To See Quarter Page Ads
Click To See Business Card Size Ads
If you have questions before you order, give me a call @ 951-235-3518 or email @ canyoncrestnewspaper@gmail.com
Like us on Facebook Here
What should Google rank in Search when all the content sucks?
Many people have an incredibly low bar for what they consider to be “great” content that...
Google Search Wants To Reward The Best Content No Matter The Size Of The Site
For the past 20 years, probably even longer, the debate about Google giving preferential...
0 Comments