Pages that search engines cannot index

What Is This Issue

This issue performs an aggregate indexability check on critical pages to confirm they are indexable by search engines. It combines multiple factors: noindex meta tags, robots.txt disallow rules, and HTTP status codes to determine if important pages are properly accessible to search engines.

A passing implementation means:

• Critical pages (homepage, main category pages, key product/service pages) return HTTP 200 status
• These pages do NOT have noindex meta tags
• These pages are NOT disallowed in robots.txt
• These pages are accessible and indexable by search engines

Example of indexable page:

• URL: https://example.com/important-page
• HTTP Status: 200 OK
• Meta robots: <meta name="robots" content="index, follow"> (or not present)
• robots.txt: Not disallowed ✅

Example of non-indexable page:

• URL: https://example.com/important-page
• HTTP Status: 200 OK
• Meta robots: <meta name="robots" content="noindex"> ❌
• Or robots.txt: Disallow: /important-page ❌

Why Is This Important

Search Visibility: If critical pages are not indexable, they won’t appear in search results, resulting in zero organic traffic from those pages.

Business Impact: The most important pages on your site (homepage, main product pages, key conversion pages) must be indexable to drive business results.

Combined Barriers: A page can be blocked from indexing in multiple ways simultaneously (noindex + robots.txt disallow + 404 status). This check identifies all barriers in one audit.

Crawl Budget Optimization: Ensuring critical pages are indexable while non-critical pages are not helps search engines focus their crawl budget on valuable content.

SEO Health Score: This aggregate check is a fundamental validation that significantly impacts the overall technical SEO health score.

How XeoPix Detects This

XeoPix performs the following aggregate checks:

1. Identifies critical pages - The system identifies important pages based on:

   • Homepage (always checked)
   • Pages with high internal link counts
   • Pages specified in sitemaps
   • Pages with high traffic or conversion value

2. Checks HTTP status - For each critical page, XeoPix verifies the HTTP status code is 200 OK.

3. Checks meta robots - XeoPix parses the HTML and looks for noindex in:

   • <meta name="robots" content="noindex"> tags
   • X-Robots-Tag: noindex HTTP headers

4. Checks robots.txt - XeoPix tests if the page URL matches any Disallow pattern in robots.txt.

5. Aggregates results - XeoPix combines all three checks and flags pages that are blocked by any barrier.

6. Reports issues - The issue is flagged if any critical page is:

   • Returning non-200 status (CRITICAL)
   • Having noindex tag (CRITICAL)
   • Blocked by robots.txt (CRITICAL)

How To Fix

1. Identify critical pages - Determine which pages are essential for SEO and business goals:

   • Homepage
   • Main category/section pages
   • Key product/service pages
   • Conversion pages (contact, pricing, etc.)

2. Check each barrier - For each critical page, verify:

   a. HTTP Status: Page should return 200 OK (not 404, 500, etc.)

   b. Meta Robots: Page should NOT have noindex:

   <!-- Remove this if present on critical pages -->
   <meta name="robots" content="noindex" />

   c. Robots.txt: Page should NOT be disallowed:

   # Remove this rule if it blocks critical pages
   Disallow: /important-page

3. Fix issues - Depending on what’s blocking indexability:

   • Remove noindex meta tags from critical pages
   • Remove Disallow rules that block critical pages
   • Fix server errors (500) or missing pages (404)

4. Validate with Search Console - Use the URL Inspection tool to verify Google sees the page as indexable.

5. Monitor regularly - Set up alerts or regular audits to ensure critical pages remain indexable.

What We Store

Storage Level

Page Level — This issue is evaluated for each individual page.

Database Table / Prisma Model

PageSeoBasicsData

Stored Fields

Detection Dependencies

• The following data sources are required to evaluate this issue:
• HTTP Response — The crawler checks the HTTP status code (200 OK required)
• robots.txt — The crawler checks if the URL is allowed by robots.txt rules
• HTML Document — The crawler checks for <meta name="robots" content="noindex"> tag

Examples

Example 1: Critical Page with noindex

Problematic State (Fails): Homepage has noindex tag:

<head>
  <meta name="robots" content="noindex" />
</head>

This prevents the homepage from being indexed.

Corrected State (Passes): Remove noindex tag from critical pages:

<head>
  <!-- No robots meta tag, or explicitly set to index -->
  <meta name="robots" content="index, follow" />
</head>

Example 2: Critical Page Blocked by robots.txt

Problematic State (Fails): Main product page is blocked:

# robots.txt
Disallow: /products/main-product

Corrected State (Passes): Remove the disallow rule for critical pages:

# robots.txt
# Don't block critical product pages
Allow: /products/main-product

Example 3: Multiple Barriers

Problematic State (Fails): Critical page has multiple indexability barriers:

• Returns 200 OK
• Has <meta name="robots" content="noindex">
• Is disallowed in robots.txt

Corrected State (Passes): Remove all barriers:

• Keep 200 OK status
• Remove noindex meta tag
• Remove Disallow rule from robots.txt

References

• Block Indexing with noindex  — Google Search Central
• Robots.txt Introduction  — Google Search Central
• HTTP Response Status Codes  — MDN