No GPTBot rule in robots.txt

What Is This Issue

This issue checks whether a website has an explicit rule in robots.txt for GPTBot, the web crawler used by OpenAI to collect data for training ChatGPT and other AI models. Site owners should make a deliberate decision to allow or disallow this bot.

A passing implementation means:

The robots.txt file contains an explicit rule for GPTBot (either Allow or Disallow)
The rule is clearly defined and intentional
If disallowed, the site’s content won’t be used for ChatGPT training

Example (Allow):


User-agent: GPTBot
Allow: /

Example (Disallow):


User-agent: GPTBot
Disallow: /

Why Is This Important

AI Training Data Control: GPTBot crawls websites to collect training data for OpenAI’s language models. Having an explicit rule lets you control whether your content is used for AI training.

Content Protection: If you don’t want your content to be used in AI training datasets, you should explicitly disallow GPTBot.

AI Visibility: Allowing GPTBot means your content may be referenced or cited in ChatGPT responses, potentially driving traffic to your site.

Legal and Ethical Considerations: Some publishers prefer to opt-out of AI training to protect their intellectual property or comply with content licensing policies.

SEO Health Score: Having explicit AI bot rules demonstrates awareness of emerging AI search technologies and improves the technical SEO score.

How XeoPix Detects This

XeoPix performs the following checks:

Fetches robots.txt - The crawler requests the /robots.txt file from the domain being audited.
Parses the content - XeoPix reads through the robots.txt file and identifies all User-agent: blocks.
Looks for GPTBot - The system searches for a User-agent: line that exactly matches GPTBot (case-sensitive).
Validates the rule - If a GPTBot user-agent block is found, XeoPix checks that it contains either:
- Disallow: directive (with any value)
- Allow: directive (with any value)
Issue Identification: XeoPix raises issues when:
- No explicit rule for GPTBot is found (SUGGESTION)
- The rule syntax is invalid (WARNING)

How To Fix

Decide your policy - Determine whether you want OpenAI to use your content for training:
- Allow: Your content may appear in ChatGPT responses and training data
- Disallow: Your content won’t be used for ChatGPT training
Add the rule to robots.txt - Add the following to your robots.txt file:
```
User-agent: GPTBot
Disallow: /
```
Or to allow:
```
User-agent: GPTBot
Allow: /
```
Place it correctly - The rule can be placed anywhere in robots.txt, but many prefer to group AI bot rules together.
Test the rule - Verify the syntax is correct using Google Search Console’s robots.txt tester or by checking the file directly.
Consider partial rules - You can disallow specific sections while allowing others:
```
User-agent: GPTBot
Disallow: /private/
Allow: /
```

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

Field	Type	Description
robotsTxtData	Json?	Parsed robots.txt content including all directives

Detection Dependencies

The following data sources are required to evaluate this issue:
robots.txt — The crawler fetches and parses the robots.txt file
HTTP Response — The crawler checks for robots.txt at https://domain.com/robots.txt

Examples

Example 1: Disallowing GPTBot

Problematic State (Fails): No explicit rule for GPTBot in robots.txt. OpenAI may crawl your site by default.

Corrected State (Passes): Add explicit disallow rule:


User-agent: GPTBot
Disallow: /

Example 2: Allowing GPTBot

Problematic State (Fails): No explicit rule for GPTBot. You want your content to be used for ChatGPT training.

Corrected State (Passes): Add explicit allow rule:


User-agent: GPTBot
Allow: /

Example 3: Partial Allow

Problematic State (Fails): You want to allow GPTBot for most content but block private pages.

Corrected State (Passes): Use partial rules:


User-agent: GPTBot
Disallow: /private/
Disallow: /admin/
Allow: /

References

GPTBot Documentation — OpenAI
Robots.txt Introduction — Google Search Central