Canonical URL blocked by robots.txt

What Is This Issue

This issue checks whether the canonical URL specified on a page is accessible to search engine crawlers and not blocked by the robots.txt file. If a canonical page is blocked, search engines cannot validate and honor the canonical directive.

A passing implementation means:

• The canonical URL is NOT disallowed in robots.txt
• The canonical URL returns a 200 OK status (not blocked, not redirected)
• Search engines can crawl the canonical URL to validate the canonical relationship

Example:

• Page URL: https://example.com/product?color=blue
• Canonical URL: https://example.com/product
• robots.txt: Disallow: /product ❌ This would block the canonical!
• Correct: No disallow for /product ✅

Why Is This Important

Canonical Validation Failure: Google explicitly states that if the canonical URL is blocked by robots.txt, they cannot crawl it and therefore cannot validate the canonical relationship. The canonical directive may be ignored.

Wrong Canonical Selection: Without access to the canonical URL, search engines may choose a different canonical than intended, potentially indexing the wrong version of the page.

Indexing Issues: Pages with blocked canonicals may not be indexed properly, or may be indexed with the wrong URL.

Wasted Canonical Tags: If the canonical URL is blocked, the canonical tag is effectively useless and provides no SEO benefit.

SEO Health Score: Resolving this issue is critical for proper canonical implementation and significantly improves the technical SEO score.

How XeoPix Detects This

XeoPix performs the following checks:

1. Extracts canonical URL - For each page crawled, XeoPix extracts the canonical URL from the <link rel="canonical"> tag.

2. Fetches robots.txt - The system retrieves and parses the site’s robots.txt file.

3. Checks if canonical is blocked - XeoPix tests whether the canonical URL matches any Disallow pattern in robots.txt.

4. Validates accessibility - The system also checks if the canonical URL:

   • Returns a 200 OK status
   • Is not redirected
   • Is not noindexed

5. Reports issues - If the canonical URL is blocked by robots.txt or inaccessible, the issue is flagged with details about which rule is blocking it.

How To Fix

1. Identify canonical URLs - Review your pages to determine what canonical URLs are specified in the <link rel="canonical"> tags.

2. Check robots.txt - Examine your robots.txt file to see if any canonical URLs or their parent paths are disallowed.

3. Remove blocking rules - If a canonical URL is blocked, either:

   • Remove the Disallow directive that blocks the canonical URL, OR
   • Change the canonical URL to a non-blocked URL, OR
   • Use meta robots with noindex instead of robots.txt blocking for the non-canonical versions

4. Verify accessibility - Ensure the canonical URL:

   • Returns HTTP 200 status
   • Is not blocked by robots.txt
   • Is not noindexed
   • Is accessible to crawlers

5. Test with Search Console - Use the URL Inspection tool in Google Search Console to verify Google can crawl the canonical URL.

What We Store

Storage Level

Page Level — This issue is evaluated for each individual page.

Database Table / Prisma Model

PageSeoBasicsData

Stored Fields

Detection Dependencies

• The following data sources are required to evaluate this issue:
• HTML Document — The crawler extracts the canonical URL from <link rel="canonical"> tag
• robots.txt — The crawler checks if the canonical URL is disallowed in robots.txt
• HTTP Response — The page content is analyzed for canonical link element

Examples

Example 1: Basic Canonical Blocked

Problematic State (Fails):

• Page: https://example.com/product?color=blue
• Canonical: https://example.com/product
• robots.txt: Disallow: /product

The canonical URL is blocked, so search engines cannot validate it.

Corrected State (Passes):

• Remove the Disallow rule for the canonical URL:

User-agent: *
Disallow: /admin/
# Don't block /product

Example 2: Partial Path Blocked

Problematic State (Fails):

• Page: https://example.com/blog/post-1
• Canonical: https://example.com/blog/post-1
• robots.txt: Disallow: /blog

The entire blog section is blocked, including the canonical URL.

Corrected State (Passes):

• Allow the canonical URL path:

User-agent: *
Allow: /blog/post-1
Disallow: /blog/admin/

Example 3: Using Meta Robots Instead

Problematic State (Fails):

• Non-canonical URLs are blocked by robots.txt
• But canonical URL is also blocked

Corrected State (Passes):

• Use meta robots for non-canonical versions:

<!-- On non-canonical URL -->
<meta name="robots" content="noindex" />

• Don’t block the canonical URL in robots.txt

Unit Test

Test File

xeopix-crawling-v2/__tests__/seo-audit-checks/crawlBehaviour/issue-175-canonical-not-blocked.test.js

Purpose

This unit test validates that the checkCanonicalUrlCrawlability() function correctly determines whether a canonical URL is accessible to search engine crawlers and not blocked by robots.txt rules.

Tested Function

• parseRobotsTxt() from robots-txt-parser.js
• checkCanonicalUrlCrawlability() from robots-txt-parser.js

Issue Information

• Issue Number: 175
• Issue Code: CANONICAL_NOT_BLOCKED
• Toggle Group: crawlBehaviour

Test Scenarios

Positive Test Cases

• Canonical URL is allowed by robots.txt (Allow: /) → isCanonicalCrawlable is true, no issues

Negative Test Cases

• Canonical URL is disallowed by robots.txt (Disallow: /about) → CANONICAL_NOT_BLOCKED issue created with blockedBy set to /about

Boundary Cases

None

Edge Cases

None

Expected Outcome

Pass

A CANONICAL_NOT_BLOCKED issue is not reported when the canonical URL is allowed by the robots.txt rules.

Fail

A CANONICAL_NOT_BLOCKED issue is reported when the canonical URL is disallowed in robots.txt, with the blocking rule pattern included in the issue details.

Validation

• Verifies that crawlable canonical URLs produce no issues
• Validates detection of blocked canonical URLs with the blocking rule pattern
• Checks payload structure (isCanonicalCrawlable, details.blockedBy)

Related Production Files

• xeopix-crawling-v2/robots-txt-parser.js
• xeopix-crawling-v2/issueCodes.js

Coverage Summary

• 2 test cases (positive and negative)
• Covers the core canonical URL crawlability detection logic
• Validates issue code and blocking rule in payload

References

• Consolidate duplicate URLs  — Google Search Central
• Robots.txt and Canonicals  — Google Search Central