Skip to Content
Crawl BehaviourIssue 22

No XML sitemap found

What Is This Issue

A sitemap is an XML file that lists a website’s URLs and metadata to guide search engine crawlers. This issue checks whether a sitemap exists, is accessible, properly formatted, and contains valid URLs that align with SEO best practices.

A passing implementation requires:

  • A sitemap file accessible at the root domain or declared in robots.txt
  • Valid XML format following sitemap protocol standards
  • URLs that return 200 OK status codes
  • No conflicting signals (e.g., URLs blocked by robots.txt or marked with noindex)
  • Proper canonical URL alignment with <loc> entries

Example: A properly configured sitemap at https://example.com/sitemap.xml that lists all important pages with accurate metadata and no broken or conflicting URLs.

Why Is This Important

Sitemaps are critical for SEO because they:

  • Improve Crawlability: Help search engines discover pages that might not be found through internal links alone, especially “orphan pages”
  • Enhance Indexability: Ensure important content is considered for indexing by providing direct URL signals
  • Support Rankings: Enable faster discovery and indexing of new or updated content
  • Optimize Crawl Budget: Guide crawlers to priority pages efficiently, especially for large sites

Sitemaps directly contribute to the overall SEO health score by ensuring search engines can efficiently discover and index your content. Sites without sitemaps or with broken sitemaps may experience delayed indexing and reduced visibility in search results.

How XeoPix Detects This

XeoPix performs comprehensive sitemap validation through the following logical steps:

  1. Sitemap Discovery: XeoPix looks for your sitemap by checking robots.txt for Sitemap: directives, trying common paths like /sitemap.xml, and examining HTML <link> tags in your pages.

  2. Sitemap Fetching: If a sitemap is found, XeoPix fetches the file and determines whether it’s a standard sitemap or a sitemap index file. For index files, XeoPix fetches each child sitemap recursively.

  3. URL Validation: For each URL listed in the sitemap, XeoPix checks:

    • The HTTP status code (should be 200 OK)
    • Whether the URL is blocked by robots.txt
    • Whether the URL has a noindex directive
    • Whether the canonical URL on the page matches the sitemap URL exactly

  4. Issue Identification: XeoPix raises issues when:

    • No sitemap is discovered (CRITICAL)
    • The sitemap returns a non-200 status code (CRITICAL)
    • URLs in the sitemap are blocked or have noindex tags (WARNING)
    • Canonical URLs don’t match sitemap URLs (WARNING)
    • The sitemap isn’t declared in robots.txt (SUGGESTION)

How To Fix

  1. Ensure sitemap accessibility: Place a sitemap file at your root domain (/sitemap.xml) and ensure it returns a 200 OK status without redirects.

  2. Declare sitemap location: Add a Sitemap: directive in your robots.txt file pointing to your sitemap URL.

  3. Remove conflicting URLs: Exclude URLs from your sitemap that are blocked by robots.txt or marked with noindex tags.

  4. Split large sitemaps: If your sitemap exceeds 50,000 URLs or 50MB, split it into multiple sitemaps and use a sitemap index file.

  5. Maintain canonical consistency: Ensure every URL in your sitemap matches the canonical URL exactly (same scheme, domain, path, and trailing slash).

  6. Include media content: For important images and videos, use image and video sitemap extensions to help search engines discover this content.

  7. Validate regularly: Periodically check that all URLs in your sitemap return 200 OK status codes and aren’t blocked by robots.txt or noindex directives.

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level, not per-page.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

FieldTypeDescription
sitemapUrlsJson?Array of sitemap URLs found during the crawl
videoSitemapEntriesJson?Array of video sitemap entries (if applicable)

Detection Dependencies

  • The following data sources are required to evaluate this issue:
  • XML Sitemap — The crawler looks for sitemap files (sitemap.xml, sitemap_index.xml, etc.)
  • Robots.txt — The crawler checks for Sitemap directives in robots.txt
  • HTTP Response — The crawler fetches and parses sitemap files

Examples

Example 1: Missing Sitemap

Problematic State (Fails): A website has no sitemap file at /sitemap.xml and no Sitemap: directive in robots.txt. Search engines must discover all pages through internal links, which may miss orphan pages.

Corrected State (Passes): Create a sitemap at https://example.com/sitemap.xml and add Sitemap: https://example.com/sitemap.xml to your robots.txt file.

Example 2: Sitemap with Broken URLs

Problematic State (Fails): A sitemap contains URLs that return 404 errors:

<url>
  <loc>https://example.com/old-page</loc>
</url>

The URL https://example.com/old-page returns a 404 Not Found status.

Corrected State (Passes): Remove broken URLs from the sitemap or fix the pages so they return 200 OK:

<url>
  <loc>https://example.com/new-page</loc>
</url>

Example 3: Sitemap with Conflicting Signals

Problematic State (Fails): A URL is in the sitemap but blocked by robots.txt:

<!-- sitemap.xml -->
<url>
  <loc>https://example.com/private-page</loc>
</url>
# robots.txt
Disallow: /private-page

Corrected State (Passes): Either remove the URL from the sitemap or remove the Disallow rule:

<!-- sitemap.xml -->
<url>
  <loc>https://example.com/private-page</loc>
</url>
# robots.txt
# Allow crawling of private-page (but keep it noindex if needed)

Unit Test

Test File

xeopix-crawling-v2/__tests__/seo-audit-checks/crawlBehaviour/issue-3-xml-sitemap-created.test.js

Purpose

This unit test validates that the checkXmlSitemapExistence() function correctly detects whether an XML sitemap exists at common locations (/sitemap.xml, /sitemap_index.xml, or custom URLs) and reports an issue when no sitemap is found.

Tested Function

checkXmlSitemapExistence() from sitemap-parser.js

Issue Information

  • Issue Number: 3
  • Issue Code: XML_SITEMAP_CREATED
  • Toggle Group: crawlBehaviour

Test Scenarios

Positive Test Cases

  • Sitemap exists at /sitemap.xml — returns 200 with valid XML content → no issue created
  • Sitemap exists at /sitemap_index.xml — first candidate returns 404, second returns 200 → no issue created
  • Sitemap exists at an alternate/custom location — all common locations fail, custom URL succeeds → no issue created
  • Empty sitemap content (status 200, empty string) → treated as existing sitemap, no issue
  • Whitespace-only sitemap content (status 200, ' \n\t ') → treated as existing sitemap, no issue

Negative Test Cases

  • No sitemap found at any location — all candidates return 404 → XML_SITEMAP_CREATED issue created with status missing

Boundary Cases

  • Timeout handling — all candidates return 404 within the timeout → no sitemap found, issue created

Edge Cases

  • Invalid seed URL — checkXmlSitemapExistence('invalid-url') → rejects with an error
  • Empty sitemap content (200 with empty string) — treated as existing (no issue)
  • Whitespace-only sitemap content — treated as existing (no issue)

Expected Outcome

Pass

An XML_SITEMAP_CREATED issue is not reported when:

  • A sitemap is found at /sitemap.xml, /sitemap_index.xml, or a custom URL that returns HTTP 200
  • The sitemap content may be empty or contain only whitespace but still returns HTTP 200

Fail

An XML_SITEMAP_CREATED issue is reported with status missing when:

  • All common and custom sitemap locations return non-200 status codes
  • The function times out after checking all locations

Validation

  • Verifies correct detection of sitemap at multiple standard locations
  • Validates fallback to custom sitemap URLs when common locations fail
  • Checks that empty/whitespace-only content is treated as valid sitemap
  • Ensures invalid URLs cause the function to reject
  • Validates timeout handling does not produce false positives
  • xeopix-crawling-v2/sitemap-parser.js
  • xeopix-crawling-v2/issueCodes.js

Coverage Summary

  • 8 test cases covering positive, negative, boundary, and edge scenarios
  • Tests multiple sitemap location resolution strategies
  • Validates both existence and absence detection paths
  • Covers empty/whitespace content edge cases
  • Covers invalid URL rejection

References

Last updated on
Issue 21Issue 23