No crawl-delay set for a rate-limited server

What Is This Issue

The Crawl-delay directive in robots.txt tells web crawlers how many seconds to wait between requests to your website. This issue checks whether a crawl-delay directive is properly set when your server needs rate limiting to prevent overload from web crawlers.

A passing implementation requires:

• A Crawl-delay directive is present in your robots.txt file
• The directive is set to a positive numeric value (e.g., Crawl-delay: 1)
• The directive is placed in the correct User-agent group
• Your robots.txt file is accessible and returns 200 OK

Example: A robots.txt file with Crawl-delay: 1 in the User-agent: * section to tell all crawlers to wait 1 second between requests.

Why Is This Important

The crawl-delay directive is important for SEO and server stability because it:

• Protects Server Resources: Prevents crawlers from overwhelming your server with too many simultaneous requests
• Maintains Site Performance: Ensures your website remains fast for real users even when being crawled
• Supports Crawl Budget Management: Helps search engines crawl your site more efficiently without causing server strain
• Prevents Downtime: Reduces the risk of server crashes due to aggressive crawling

While not all crawlers respect the crawl-delay directive, it’s a helpful signal for crawlers that do support it. This indirectly supports your SEO health score by maintaining server stability and ensuring consistent crawlability.

How XeoPix Detects This

XeoPix performs crawl-delay directive checks through the following logical steps:

1. Configuration Check: XeoPix checks if your site is configured as requiring rate limiting (based on server capacity settings).

2. Robots.txt Fetch: XeoPix attempts to fetch your robots.txt file from the root domain.

3. Directive Parsing: If robots.txt exists, XeoPix parses the file line-by-line to find Crawl-delay directives within User-agent groups.

4. Value Validation: XeoPix validates that:

   • A Crawl-delay directive exists
   • The value is a positive number
   • The directive is in the correct section

5. Issue Identification: XeoPix raises issues when:

   • Crawl-delay is missing but server needs rate limiting (SUGGESTION)
   • The value is invalid (not a number) (WARNING)
   • The directive is in the wrong section (WARNING)

How To Fix

1. Check if you need rate limiting: Evaluate whether your server experiences high load during crawls or has limited resources.

2. Edit your robots.txt file: Add a Crawl-delay directive to your robots.txt file:

   User-agent: *
   Crawl-delay: 1

   (The number represents seconds to wait between requests)

3. Choose an appropriate value: Set a reasonable delay based on your server capacity:

   • 1 second for most sites
   • 2-5 seconds for servers with limited resources
   • Higher values for very resource-constrained servers

4. Place in correct section: Ensure the Crawl-delay directive is within a User-agent group (either * for all crawlers or a specific crawler like Googlebot).

5. Test your robots.txt: Verify your robots.txt file is accessible and properly formatted.

6. Monitor server load: After implementing, monitor your server performance during crawls to ensure the delay is effective.

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

Detection Dependencies

• The following data sources are required to evaluate this issue:
• robots.txt — The crawler parses robots.txt and looks for Crawl-delay: directives
• HTTP Response — The crawler fetches robots.txt from the root domain

Examples

Example 1: Adding Crawl-delay

Problematic State (Fails): Server experiences high load during crawls, but no crawl-delay is set:

User-agent: *
Disallow: /admin/

Corrected State (Passes): Add Crawl-delay directive:

User-agent: *
Crawl-delay: 1
Disallow: /admin/

Example 2: Wrong Value

Problematic State (Fails): Crawl-delay set to an invalid value:

Crawl-delay: yes

Corrected State (Passes): Use a numeric value (seconds):

Crawl-delay: 1

Example 3: Specific Crawler

Problematic State (Fails): Crawl-delay is set for all crawlers, but you only want to limit a specific crawler:

User-agent: *
Crawl-delay: 1

Corrected State (Passes): Target specific crawler:

User-agent: Bingbot
Crawl-delay: 2

User-agent: *
Disallow: /admin/

Unit Test

Test File

xeopix-crawling-v2/__tests__/seo-audit-checks/crawlBehaviour/issue-165-crawl-delay-directive.test.js

Purpose

This unit test validates that the checkCrawlDelayDirective() function correctly detects whether a robots.txt file contains a Crawl-delay directive, reporting an issue when one is present.

Tested Function

checkCrawlDelayDirective() from robots-txt-parser.js

Issue Information

• Issue Number: 165
• Issue Code: CRAWL_DELAY_DIRECTIVE
• Toggle Group: crawlBehaviour

Test Scenarios

Positive Test Cases

• No Crawl-delay directive present in robots.txt → hasCrawlDelay is false, no issue created

Negative Test Cases

• Crawl-delay directive is declared with a value of 5 → CRAWL_DELAY_DIRECTIVE issue created with delay value of 5

Boundary Cases

None

Edge Cases

None

Expected Outcome

Pass

A CRAWL_DELAY_DIRECTIVE issue is not reported when the robots.txt file does not contain any Crawl-delay directive.

Fail

A CRAWL_DELAY_DIRECTIVE issue is reported when a Crawl-delay directive is present, with the delay value included in the issue details.

Validation

• Verifies absence of crawl-delay produces no issues
• Validates detection of crawl-delay directive with correct delay value
• Checks payload structure (hasCrawlDelay, details.delays[0].delay)

Related Production Files

• xeopix-crawling-v2/robots-txt-parser.js
• xeopix-crawling-v2/issueCodes.js

Coverage Summary

• 2 test cases (positive and negative)
• Covers the core crawl-delay detection logic
• Validates issue code and delay value in payload

References

• Robots.txt Introduction  — Google Search Central
• RFC 9309 - Robots Exclusion Protocol  — RFC Editor