robots.txt missing or misconfigured

What Is This Issue

A robots.txt file is a plain-text file placed at a website’s root directory (e.g., https://example.com/robots.txt) that instructs web crawlers which parts of the site they can or cannot access. This issue checks whether a robots.txt file exists, is accessible, properly configured, and doesn’t contain rules that block important pages from being crawled or indexed.

A passing implementation requires:

• A robots.txt file accessible at the root domain returning 200 OK
• Properly formatted directives following the Robots Exclusion Protocol
• No rules that block important pages, CSS, or JavaScript files
• No conflicting signals (e.g., blocking a URL that’s in your sitemap)
• Proper noindex directives via meta tags or HTTP headers (not in robots.txt)

Example: A properly configured robots.txt that allows all crawlers access to public pages while blocking access to private admin areas.

Why Is This Important

The robots.txt file is critical for SEO because it:

• Controls Crawlability: Determines which pages search engines can crawl and which they must avoid
• Protects Indexability: Prevents important pages from being blocked unintentionally
• Manages Crawl Budget: Helps search engines focus on your most important content
• Prevents Duplicate Content: Can block crawlers from accessing duplicate or low-value pages

Misconfigured robots.txt files can accidentally block your entire site from search engines or prevent important pages from being discovered. This directly impacts your SEO health score by affecting how search engines discover, crawl, and index your content.

How XeoPix Detects This

XeoPix performs comprehensive robots.txt analysis through the following logical steps:

1. File Discovery: XeoPix attempts to fetch the robots.txt file from your root domain and records the HTTP status code.

2. Directive Parsing: If the file exists, XeoPix parses all directives line-by-line, including:

   • User-agent rules (which crawlers the rules apply to)
   • Disallow rules (what’s blocked)
   • Allow rules (what’s explicitly allowed)
   • Sitemap declarations
   • Crawl-delay settings

3. Conflict Detection: XeoPix cross-references robots.txt rules against:

   • Your sitemap URLs (to find blocked sitemap entries)
   • Canonical URLs (to check if they’re blocked)
   • CSS and JavaScript files (to ensure they’re accessible)

4. Noindex Detection: XeoPix checks for noindex directives in:

   • HTML <meta name="robots" content="noindex"> tags
   • HTTP X-Robots-Tag response headers

5. Issue Identification: XeoPix raises issues when:

   • robots.txt is missing or returns an error (CRITICAL or WARNING)
   • Important pages or resources are blocked (CRITICAL)
   • Sitemap URLs conflict with Disallow rules (IMPORTANT)
   • Noindex directives are missing on private files (WARNING)
   • robots.txt hasn’t been updated in over a year (SUGGESTION)

How To Fix

1. Ensure robots.txt exists: Create a robots.txt file at your root domain (/robots.txt) that returns 200 OK when accessed.

2. Review Disallow rules: Check that you’re not blocking important pages, CSS, or JavaScript files that search engines need to render your pages properly.

3. Remove conflicting rules: Ensure URLs listed in your sitemap aren’t blocked by robots.txt Disallow rules.

4. Use proper noindex method: Don’t use robots.txt to prevent indexing. Instead, use:

   • <meta name="robots" content="noindex"> in your HTML
   • X-Robots-Tag: noindex in your HTTP headers

5. Declare your sitemap: Add a Sitemap: directive pointing to your XML sitemap location.

6. Don’t expose sensitive paths: Avoid listing private directories (like /admin/) in robots.txt, as it publicly reveals their existence.

7. Validate regularly: Periodically check your robots.txt file to ensure it’s accessible and properly configured.

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level, not per-page.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

Detection Dependencies

• The following data sources are required to evaluate this issue:
• robots.txt — The crawler fetches and parses the robots.txt file from the root domain
• HTTP Response — The crawler checks for robots.txt at https://domain.com/robots.txt 

Examples

Example 1: Missing robots.txt

Problematic State (Fails): A website has no robots.txt file at /robots.txt. Search engines don’t know which pages they can or cannot crawl, which may lead to crawling of private or low-value pages.

Corrected State (Passes): Create a robots.txt file at https://example.com/robots.txt:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Example 2: Blocking Important Resources

Problematic State (Fails): A robots.txt file blocks CSS and JavaScript files:

User-agent: *
Disallow: /css/
Disallow: /js/

Search engines cannot render the page properly without these resources.

Corrected State (Passes): Allow access to CSS and JavaScript files:

User-agent: *
Allow: /
# Don't block CSS or JS files
Sitemap: https://example.com/sitemap.xml

Example 3: Conflicting Sitemap Rules

Problematic State (Fails): A sitemap contains a URL that’s blocked by robots.txt:

# robots.txt
Disallow: /private-page

<!-- sitemap.xml -->
<url>
  <loc>https://example.com/private-page</loc>
</url>

Corrected State (Passes): Either remove the URL from the sitemap or remove the Disallow rule:

# robots.txt
# Allow crawling of private-page
Sitemap: https://example.com/sitemap.xml

Unit Test

Test File

xeopix-crawling-v2/__tests__/seo-audit-checks/crawlBehaviour/issue-5-robots-txt-configured.test.js

Purpose

This unit test validates the robots-txt-parser.js module, covering parseRobotsTxt(), analyzeRobotsTxt(), and fetchRobotsTxt() functions. It ensures correct parsing of valid robots.txt content, detection of missing or misconfigured files, and proper handling of edge cases.

Tested Function

• parseRobotsTxt() from robots-txt-parser.js
• analyzeRobotsTxt() from robots-txt-parser.js
• fetchRobotsTxt() from robots-txt-parser.js

Issue Information

• Issue Number: 5
• Issue Code: ROBOTS_TXT_CONFIGURED
• Toggle Group: crawlBehaviour

Test Scenarios

Positive Test Cases

• Valid robots.txt with allow rules, disallow rules, crawl-delay, and sitemap directives → parsed correctly with no ROBOTS_TXT_CONFIGURED issue
• fetchRobotsTxt() successful fetch with HTTP 200 and valid content
• analyzeRobotsTxt() with HTTP 200 and valid content → no ROBOTS_TXT_CONFIGURED issue

Negative Test Cases

• analyzeRobotsTxt() with HTTP 404 → ROBOTS_TXT_CONFIGURED issue with status not-found and httpStatus 404
• analyzeRobotsTxt() with HTTP 500 → ROBOTS_TXT_CONFIGURED issue with status server-error and httpStatus 500
• fetchRobotsTxt() with HTTP 404 → returns status 404 and null data

Boundary Cases

• robots.txt with only comments → parsed successfully, no crash
• analyzeRobotsTxt() with only comments → defined and no crash

Edge Cases

• Empty robots.txt → empty rules and sitemap URLs
• Whitespace-only robots.txt → no rules parsed
• Malformed robots.txt (random text without directives) → no rules parsed, no crash
• Wildcard patterns (/*.pdf$, /private/*) → correctly stored in disallow list
• Disallow: / for all user agents → isEntireSiteBlocked('*') returns true
• Multiple user-agent blocks → correctly parsed with 3 rule groups
• Sensitive paths exposed (/admin/, /private/) → issue detected with status sensitive-paths-exposed
• CSS/JS resources blocked (/css/, /js/) → issue detected with status css-js-blocked
• fetchRobotsTxt() with network error → returns status 0 and null data

Expected Outcome

Pass

A ROBOTS_TXT_CONFIGURED issue is not reported when:

• The robots.txt file returns HTTP 200 with valid content
• The file contains proper directives and is accessible

Fail

A ROBOTS_TXT_CONFIGURED issue is reported when:

• robots.txt returns HTTP 404 (not-found)
• robots.txt returns HTTP 500 (server-error)
• robots.txt exposes sensitive paths (sensitive-paths-exposed)
• robots.txt blocks CSS/JS resources (css-js-blocked)

Validation

• Verifies correct parsing of all standard robots.txt directives (Allow, Disallow, Crawl-delay, Sitemap)
• Validates HTTP status-based detection (200, 404, 500)
• Checks for sensitive path exposure and CSS/JS blocking detection
• Ensures resilience with empty, whitespace-only, and malformed input
• Validates multiple user-agent block handling
• Tests network error resilience in fetchRobotsTxt()

Related Production Files

• xeopix-crawling-v2/robots-txt-parser.js
• xeopix-crawling-v2/issueCodes.js

Coverage Summary

• 15 test cases across three function groups (parseRobotsTxt, analyzeRobotsTxt, fetchRobotsTxt)
• Covers positive, negative, boundary, and edge scenarios
• Tests HTTP error handling (404, 500, network errors)
• Validates content parsing (empty, whitespace, malformed, wildcard patterns)
• Covers security detection (sensitive paths, CSS/JS blocking)

References

• Robots.txt Introduction  — Google Search Central
• RFC 9309 - Robots Exclusion Protocol  — RFC Editor
• Block Indexing with noindex  — Google Search Central