No ClaudeBot rule in robots.txt

What Is This Issue

This issue checks whether a website has an explicit rule in robots.txt for ClaudeBot, the web crawler used by Anthropic to collect data for training Claude AI models. Site owners should make a deliberate decision to allow or disallow this bot.

A passing implementation means:

The robots.txt file contains an explicit rule for ClaudeBot (either Allow or Disallow)
The rule is clearly defined and intentional
If disallowed, the site’s content won’t be used for Claude training

Example (Allow):


User-agent: ClaudeBot
Allow: /

Example (Disallow):


User-agent: ClaudeBot
Disallow: /

Why Is This Important

AI Training Data Control: ClaudeBot crawls websites to collect training data for Anthropic’s Claude AI models. Having an explicit rule lets you control whether your content is used for AI training.

Content Protection: If you don’t want your content to be used in Anthropic’s AI training datasets, you should explicitly disallow ClaudeBot.

AI Visibility: Allowing ClaudeBot means your content may be referenced or cited in Claude responses, potentially driving traffic to your site.

Legal and Ethical Considerations: Some publishers prefer to opt-out of AI training to protect their intellectual property or comply with content licensing policies.

Emerging Standard: As AI crawlers become more prevalent, having explicit rules demonstrates technical sophistication and awareness of the evolving search landscape.

SEO Health Score: Having explicit AI bot rules demonstrates awareness of emerging AI search technologies and improves the technical SEO score.

How XeoPix Detects This

XeoPix performs the following checks:

Fetches robots.txt - The crawler requests the /robots.txt file from the domain being audited.
Parses the content - XeoPix reads through the robots.txt file and identifies all User-agent: blocks.
Looks for ClaudeBot - The system searches for a User-agent: line that exactly matches ClaudeBot (case-sensitive).
Validates the rule - If a ClaudeBot user-agent block is found, XeoPix checks that it contains either:
- Disallow: directive (with any value)
- Allow: directive (with any value)
Issue Identification: XeoPix raises issues when:
- No explicit rule for ClaudeBot is found (SUGGESTION)
- The rule syntax is invalid (WARNING)

How To Fix

Decide your policy - Determine whether you want Anthropic to use your content for training Claude:
- Allow: Your content may appear in Claude responses and training data
- Disallow: Your content won’t be used for Claude training
Add the rule to robots.txt - Add the following to your robots.txt file:
```
User-agent: ClaudeBot
Disallow: /
```
Or to allow:
```
User-agent: ClaudeBot
Allow: /
```
Place it correctly - The rule can be placed anywhere in robots.txt, but many prefer to group AI bot rules together.
Test the rule - Verify the syntax is correct using Google Search Console’s robots.txt tester or by checking the file directly.
Consider partial rules - You can disallow specific sections while allowing others:
```
User-agent: ClaudeBot
Disallow: /private/
Allow: /
```

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

Field	Type	Description
robotsTxtData	Json?	Parsed robots.txt content including all directives

Detection Dependencies

The following data sources are required to evaluate this issue:
robots.txt — The crawler fetches and parses the robots.txt file to check for syntax errors
HTTP Response — The crawler checks for robots.txt at https://domain.com/robots.txt

Examples

Example 1: Disallowing ClaudeBot

Problematic State (Fails): No explicit rule for ClaudeBot in robots.txt. Anthropic may crawl your site by default.

Corrected State (Passes): Add explicit disallow rule:


User-agent: ClaudeBot
Disallow: /

Example 2: Allowing ClaudeBot

Problematic State (Fails): No explicit rule for ClaudeBot. You want your content to be used for Claude training.

Corrected State (Passes): Add explicit allow rule:


User-agent: ClaudeBot
Allow: /

Example 3: Partial Allow

Problematic State (Fails): You want to allow ClaudeBot for most content but block private pages.

Corrected State (Passes): Use partial rules:


User-agent: ClaudeBot
Disallow: /private/
Disallow: /admin/
Allow: /

References

ClaudeBot Documentation — Anthropic
Robots.txt Introduction — Google Search Central