No CCBot rule in robots.txt

What Is This Issue

This issue checks whether a website has an explicit rule in robots.txt for CCBot, the web crawler operated by Common Crawl. Common Crawl maintains an open repository of web crawl data that powers many open-source LLMs and AI research projects. Site owners should make an explicit decision to allow or disallow this bot.

A passing implementation means:

The robots.txt file contains an explicit rule for CCBot (either Allow or Disallow)
The rule is clearly defined and intentional
Site owners have decided whether to contribute to the Common Crawl dataset

Example (Allow):


User-agent: CCBot
Allow: /

Example (Disallow):


User-agent: CCBot
Disallow: /

Why Is This Important

Open-Source AI Training: Common Crawl data is used to train many open-source Large Language Models (LLMs) like GPT-J, GPT-Neo, and others. Your content may contribute to these models if you allow CCBot.

Research and Archiving: Common Crawl provides web data for researchers, archivists, and non-commercial projects. Allowing CCBot supports open web data initiatives.

Content Control: If you don’t want your content to be part of the Common Crawl dataset (and subsequently used in open-source AI training), you should explicitly disallow CCBot.

Public Dataset: Unlike some proprietary crawlers, Common Crawl data is publicly accessible, meaning your content could be used by anyone with access to the dataset.

Ethical Considerations: Some publishers prefer to opt-out of large-scale web scraping for AI training to protect their intellectual property or control how their content is used.

SEO Health Score: Having explicit AI bot rules demonstrates awareness of emerging AI search technologies and improves the technical SEO score.

How XeoPix Detects This

XeoPix performs the following checks:

Fetches robots.txt - The crawler requests the /robots.txt file from the domain being audited.
Parses the content - XeoPix reads through the robots.txt file and identifies all User-agent: blocks.
Looks for CCBot - The system searches for a User-agent: line that exactly matches CCBot (case-sensitive).
Validates the rule - If a CCBot user-agent block is found, XeoPix checks that it contains either:
- Disallow: directive (with any value)
- Allow: directive (with any value)
Issue Identification: XeoPix raises issues when:
- No explicit rule for CCBot is found (SUGGESTION)
- The rule syntax is invalid (WARNING)

How To Fix

Decide your policy - Determine whether you want your content to be part of the Common Crawl dataset:
- Allow: Your content will be part of the open web crawl dataset used for research and open-source AI
- Disallow: Your content won’t be included in Common Crawl’s dataset
Add the rule to robots.txt - Add the following to your robots.txt file:
```
User-agent: CCBot
Disallow: /
```
Or to allow:
```
User-agent: CCBot
Allow: /
```
Place it correctly - The rule can be placed anywhere in robots.txt, but many prefer to group AI bot rules together.
Test the rule - Verify the syntax is correct using Google Search Console’s robots.txt tester or by checking the file directly.
Consider partial rules - You can disallow specific sections while allowing others:
```
User-agent: CCBot
Disallow: /private/
Allow: /
```

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

Field	Type	Description
robotsTxtData	Json?	Parsed robots.txt content including all directives

Detection Dependencies

The following data sources are required to evaluate this issue:
robots.txt — The crawler fetches and parses the robots.txt file to check for user-agent directives
HTTP Response — The crawler checks for robots.txt at https://domain.com/robots.txt

Examples

Example 1: Disallowing CCBot

Problematic State (Fails): No explicit rule for CCBot in robots.txt. Common Crawl may crawl your site by default.

Corrected State (Passes): Add explicit disallow rule:


User-agent: CCBot
Disallow: /

Example 2: Allowing CCBot

Problematic State (Fails): No explicit rule for CCBot. You want your content to be part of the open dataset.

Corrected State (Passes): Add explicit allow rule:


User-agent: CCBot
Allow: /

Example 3: Grouping All AI Bots

Problematic State (Fails): AI bot rules are scattered throughout robots.txt.

Corrected State (Passes): Group all AI bot rules together:


# Search engine crawlers
User-agent: Googlebot
Allow: /

# AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

References

Common Crawl Documentation — Common Crawl
Robots.txt Introduction — Google Search Central