robots.txt does not point to the sitemap

What Is This Issue

This issue checks whether a website’s robots.txt file includes a Sitemap: directive that explicitly declares the location of the XML sitemap, providing an additional method for search engines to discover the sitemap.

A passing implementation means:

The robots.txt file exists at /robots.txt
It contains one or more Sitemap: directives pointing to valid sitemap URLs
The sitemap URLs are absolute and properly formatted
Multiple sitemaps (if applicable) are all listed

Example:


User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xml

Why Is This Important

Sitemap Discovery: While search engines can discover sitemaps through other means (Google Search Console, sitemap index files), the Sitemap: directive in robots.txt is a standard method recognized by all major crawlers including Google, Bing, Yahoo, and others.

Crawler Efficiency: Search engines often check robots.txt first before crawling. Having the sitemap directive there allows them to discover and process the sitemap immediately.

Backup Discovery Method: If sitemaps are not submitted through Search Console or are not linked from the homepage, the robots.txt directive serves as a fallback discovery mechanism.

SEO Best Practice: Including sitemap locations in robots.txt is a widely recommended SEO practice that demonstrates technical optimization.

SEO Health Score: Having the Sitemap: directive improves the technical SEO score by showing comprehensive sitemap accessibility.

How XeoPix Detects This

XeoPix performs the following checks:

Fetches robots.txt - The crawler requests the /robots.txt file from the domain being audited.
Parses the content - XeoPix reads through the robots.txt file line by line.
Looks for Sitemap directives - The system searches for lines starting with Sitemap: (case-insensitive).
Validates the URLs - For each Sitemap: directive found, XeoPix:
- Checks that the URL is absolute (starts with http:// or https://)
- Attempts to fetch the sitemap to verify it’s accessible
- Validates the sitemap format
Issue Identification: XeoPix raises issues when:
- No Sitemap: directive is found in robots.txt (SUGGESTION)
- The sitemap URL is relative instead of absolute (WARNING)
- The sitemap URL returns an error (WARNING)
- The Sitemap: directive is malformed (WARNING)

How To Fix

Locate your robots.txt file - This should be at the root of your domain (https://example.com/robots.txt).
Add Sitemap directive(s) - Add one line per sitemap at the end of the robots.txt file:
```
Sitemap: https://example.com/sitemap.xml
```
Use absolute URLs - Always use the full absolute URL including https://, not relative paths.
List all sitemaps - If you have multiple sitemaps (main, images, videos, etc.), list each one on a separate line.
Validate the format - Ensure there are no spaces before Sitemap: and the colon is immediately followed by the URL.
Test accessibility - Verify the robots.txt file is accessible and properly formatted by visiting https://example.com/robots.txt in a browser.

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

Field	Type	Description
hasRobotsTxtSitemapDirective	Boolean	Whether robots.txt contains a Sitemap directive

Detection Dependencies

The following data sources are required to evaluate this issue:
robots.txt — The crawler parses robots.txt and looks for Sitemap: directives
HTTP Response — The crawler fetches robots.txt from the root domain

Examples

Example 1: Missing Sitemap Directive

Problematic State (Fails): robots.txt exists but doesn’t declare the sitemap:


User-agent: *
Disallow: /admin/

Corrected State (Passes): Add the Sitemap directive:


User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Example 2: Multiple Sitemaps

Problematic State (Fails): Only the main sitemap is declared, but image sitemap is missing:


User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Corrected State (Passes): Declare all sitemaps:


User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xml

Example 3: Relative URL

Problematic State (Fails): Using relative URL instead of absolute:


Sitemap: /sitemap.xml

Corrected State (Passes): Use absolute URL:


Sitemap: https://example.com/sitemap.xml

Unit Test

Test File

xeopix-crawling-v2/__tests__/seo-audit-checks/crawlBehaviour/issue-164-sitemap-directive.test.js

Purpose

This unit test validates that the checkSitemapDirective() function correctly detects whether a robots.txt file declares a Sitemap: directive, reporting an issue when no sitemap directive is present.

Tested Function

checkSitemapDirective() from robots-txt-parser.js

Issue Information

Issue Number: 164
Issue Code: SITEMAP_DIRECTIVE_DECLARED
Toggle Group: crawlBehaviour

Test Scenarios

Positive Test Cases

robots.txt declares a sitemap URL → hasSitemap is true, no issue created

Negative Test Cases

No sitemap directive present (empty array) → SITEMAP_DIRECTIVE_DECLARED issue created with status no-sitemap-directive

Boundary Cases

None

Edge Cases