Google Clarifies the “Google-Extended” Crawler Documentation

Feb 9, 2024 | SEO News Feeds | 0 comments

SEO Content Writing Service

Google recently updated the documentation of its Google-Extended web crawler user agent, reflecting changes in product naming and clarifying the impact on search, which may be a concern for those who choose to block the crawler. The updated documentation offers clearer guidance on controlling content access for use in AI model training.

Google-Extended User Agent

Introduced on September 28, 2023, Google-Extended offers web publishers a user agent that can be used to control how their sites are crawled. Publishers can allow or disallow the Google-Extended user agent using the Robots Exclusion Protocol, giving them a way to opt-out of having their content scraped and included in AI training datasets.

Google describes Google-Extended as a “standalone product token” but that’s non-standard terminology for how publishers understand the concept of User Agents.

The original announcement described the new user agent:

“Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products.

By using Google-Extended to control access to content on a site, a website administrator can choose whether to help these AI models become more accurate and capable over time.”

Blocking Google-Extended is done with the “Google-Extended” User Agent:

User-agent: Google-Extended
Disallow: /

Google Changelog

Google keeps a changelog of important updates made to guidance and communication with web publishers and the search marketing community. The changelog of Google’s developer pages announced a change to the Google-Extended documentation.

The revision comes after the renaming of Bard to Gemini Apps, specifying that Google-Extended’s indexing now contributes to Gemini Apps and Vertex AI generative APIs. The new wording reassures publishers that this does not affect Google Search, addressing potential concerns about the possible implications from opting out of Google-Extended AI data collection.

What Changed?

Google’s changelog clarifies that Google-Extended crawling is exclusive to Gemini Apps and has no impact on Google Search.

Attorney Websites For Sale 4ebusiness Media Group

The Changelog advises:

“Updated the description of the Google-Extended product token
What: With the name change of Bard to Gemini Apps, we clarified that Gemini Apps is affected by Google-Extended, and, based on publisher feedback, we specified that Google-Extended doesn’t affect Google Search.”

The updated guidance no longer uses the Bard brand name, switching it out to Gemini. And the following sentence was added:

“Google-Extended does not impact a site’s inclusion or ranking in Google Search.”

Read Google’s updated crawler overview:

Overview of Google crawlers and fetchers (user agents)

Featured Image by Shutterstock/Ribkhan

Source link

Anxiety Stress Management

Live a Life of Contentment eBook We all want to be satisfied, even though we know some people who will never be that way, and others who see satisfaction as a foreign emotion that they can’t hope to ever feel.

Newspaper Ads Canyon Crest CA

Click To See Full Page Ads

Click To See Half Page Ads

Click To See Quarter Page Ads

Click To See Business Card Size Ads

If you have questions before you order, give me a call @ 951-235-3518 or email @ Like us on Facebook Here

You May Also Like


Submit a Comment

Your email address will not be published. Required fields are marked *

Contact Us

Contact Us

Personal Injury Attorney

Websites For Sale Personal Injury Attorneys

Criminal Defense Attorneys

Websites For Sale Criminal Defense Attorney

Bankruptcy Attorneys

Websites For Sale Bankruptcy Attorneys

General Practice Attorneys

Websites For Sale General Practice Attorneys

Family Attorneys

Websites For Sale Family Attorneys

Corporate Attorneys

Websites For Sale Corporate Attorneys

Home Privacy Policy Terms Of Use Anti Spam Policy Contact Us Affiliate Disclosure Amazon Affiliate Disclaimer DMCA Earnings Disclaimer