No Google-Extended rule in robots.txt

What Is This Issue

This issue checks whether a website has an explicit rule in robots.txt for Google-Extended, a separate user-agent from Googlebot that controls whether content is used for Google’s generative AI products (like Bard/Gemini). This is different from regular Googlebot which crawls for search indexing.

A passing implementation means:

The robots.txt file contains an explicit rule for Google-Extended (either Allow or Disallow)
The rule is clearly defined and intentional
Site owners have made a deliberate choice about contributing content to Google’s AI training

Example (Allow):


User-agent: Google-Extended
Allow: /

Example (Disallow):


User-agent: Google-Extended
Disallow: /

Why Is This Important

Separate from Googlebot: Google-Extended is a distinct crawler from regular Googlebot. Blocking or allowing Googlebot does NOT affect Google-Extended, and vice versa.

Generative AI Training: Google-Extended is used to crawl content for Google’s generative AI products (Bard/Gemini). This is content used for AI model training, not regular search indexing.

Content Control: Having an explicit rule lets you control whether your content contributes to Google’s AI training datasets while still allowing regular Google search indexing.

Strategic Decision: Publishers must decide whether to contribute content to AI training:

Allow: Support AI development, potential citation in AI responses
Disallow: Protect content from being used in AI training datasets

Emerging Standard: As Google expands its generative AI offerings, controlling Google-Extended is becoming an important part of robots.txt management.

SEO Health Score: Having explicit AI bot rules demonstrates awareness of emerging AI search technologies and improves the technical SEO score.

How XeoPix Detects This

XeoPix performs the following checks:

Fetches robots.txt - The crawler requests the /robots.txt file from the domain being audited.
Parses the content - XeoPix reads through the robots.txt file and identifies all User-agent: blocks.
Looks for Google-Extended - The system searches for a User-agent: line that matches Google-Extended (case-sensitive).
Validates the rule - If a Google-Extended user-agent block is found, XeoPix checks that it contains either:
- Disallow: directive (with any value)
- Allow: directive (with any value)
Issue Identification: XeoPix raises issues when:
- No explicit rule for Google-Extended is found (SUGGESTION)
- The rule syntax is invalid (WARNING)

How To Fix

Decide your policy - Determine whether you want Google to use your content for generative AI training:
- Allow: Your content may be used in Google’s AI products (Bard/Gemini)
- Disallow: Your content won’t be used for Google’s AI training
Note the distinction - Remember:
- Googlebot = regular search indexing (usually want to allow)
- Google-Extended = generative AI training (separate decision)
Add the rule to robots.txt - Add the following to your robots.txt file:
```
User-agent: Google-Extended
Disallow: /
```
Or to allow:
```
User-agent: Google-Extended
Allow: /
```
Test the rule - Verify the syntax is correct using Google Search Console’s robots.txt tester or by checking the file directly.
Consider partial rules - You can disallow specific sections while allowing others:
```
User-agent: Google-Extended
Disallow: /private/
Allow: /
```

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

Field	Type	Description
robotsTxtData	Json?	Parsed robots.txt content including all directives

Detection Dependencies

The following data sources are required to evaluate this issue:
robots.txt — The crawler fetches and parses the robots.txt file to check for wildcard patterns
HTTP Response — The crawler checks for robots.txt at https://domain.com/robots.txt

Examples

Example 1: Disallowing Google-Extended

Problematic State (Fails): No explicit rule for Google-Extended in robots.txt. Google may use your content for AI training by default.

Corrected State (Passes): Add explicit disallow rule:


User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

Example 2: Allowing Google-Extended

Problematic State (Fails): No explicit rule for Google-Extended. You want your content to be used for Google’s AI products.

Corrected State (Passes): Add explicit allow rule:


User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

Example 3: Partial Allow

Problematic State (Fails): You want to allow Google-Extended for most content but block private pages.

Corrected State (Passes): Use partial rules:


User-agent: Google-Extended
Disallow: /private/
Disallow: /admin/
Allow: /

References

Google-Extended Documentation — Google Search Central
Robots.txt Introduction — Google Search Central