Internal search pages left crawlable

What Is This Issue

This issue checks whether a website’s internal search results pages are blocked from search engine crawlers using the robots.txt file or meta robots tags, preventing crawl budget waste and duplicate content issues.

A passing implementation means:

Search results pages (e.g., /search?q=..., /results?...) are disallowed in robots.txt OR have noindex meta tags
The block is properly implemented and covers all search result URL patterns
Faceted search and filtered results pages are also blocked if applicable

Example robots.txt:


User-agent: *
Disallow: /search
Disallow: /results
Disallow: /search-results

Example meta robots tag:


<meta name="robots" content="noindex, nofollow" />

Why Is This Important

Crawl Budget Waste: Search results pages often generate infinite URL variations with minimal unique content. Crawling them wastes valuable crawl budget that should be spent on important pages.

Duplicate Content: Search results pages often contain snippets of content from other pages, creating near-duplicate content that can confuse search engines.

Poor User Experience in SERPs: Search results pages in search results provide a poor user experience - users expect to land on actual content, not more search results.

Thin Content: Search results pages are typically thin content pages that don’t provide value to search engine users.

SEO Health Score: Blocking search results pages improves the technical SEO score by preventing crawl budget waste and duplicate content issues.

How XeoPix Detects This

XeoPix performs the following checks:

URL Pattern Detection: XeoPix identifies search result pages by looking for common URL patterns:
- /search?q=...
- /results?...
- /search-results
- Other custom search URL patterns
Robots.txt Check: XeoPix checks if these URL patterns are disallowed in robots.txt.
Meta Robots Check: For pages that are accessible, XeoPix checks if they have noindex meta tags.
Faceted Navigation Check: XeoPix also checks for faceted search URLs with multiple parameters.
Issue Identification: XeoPix raises issues when:
- Search result pages are not blocked in robots.txt (WARNING)
- Search result pages don’t have noindex tags (WARNING)
- Faceted navigation creates infinite URL variations (SUGGESTION)

How To Fix

Identify search result URLs - Determine the URL patterns for your site’s search functionality (e.g., /search?q=, /results?, /search-results).
Block in robots.txt - Add Disallow directives for search result URL patterns:
```
Disallow: /search
Disallow: /results
```
Alternative: Use meta robots - If you can’t block in robots.txt, add <meta name="robots" content="noindex"> to search result pages.
Check faceted navigation - If your site has faceted search (filters, sorting), ensure those URLs are also blocked:
```
Disallow: /search?
Disallow: /products/filter/
```
Test the block - Use Google Search Console’s robots.txt tester to verify the block is working.
Monitor indexing - Check Google Search Console to ensure search result pages are not appearing in the index.

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

Field	Type	Description
hasSearchPagesDisallowedInRobots	Boolean	Whether robots.txt disallows search result pages

Detection Dependencies

The following data sources are required to evaluate this issue:
robots.txt — The crawler parses robots.txt and checks for disallow rules targeting search pages
HTTP Response — The crawler fetches robots.txt from the root domain

Examples

Example 1: Basic Search Results Block

Problematic State (Fails): Search results pages are crawlable:


https://example.com/search?q=product

Returns 200 OK and is indexed by search engines.

Corrected State (Passes): Block in robots.txt:


User-agent: *
Disallow: /search

Example 2: Faceted Search URLs

Problematic State (Fails): Faceted navigation creates infinite URLs:


https://example.com/products?color=red&size=large&brand=nike
https://example.com/products?size=large&brand=nike&color=red

These create duplicate content issues.

Corrected State (Passes): Block parameterized URLs:


User-agent: *
Disallow: /products?

Example 3: Using Meta Robots

Problematic State (Fails): Can’t block in robots.txt, but search pages are indexed.

Corrected State (Passes): Add noindex meta tag to search result pages:


<head>
  <meta name="robots" content="noindex, nofollow" />
</head>

Unit Test

Test File

xeopix-crawling-v2/__tests__/seo-audit-checks/crawlBehaviour/issue-166-search-results-blocked.test.js

Purpose

This unit test validates that the checkInternalSearchPagesBlocked() function correctly detects whether internal search result pages are blocked by robots.txt, preventing crawl budget waste and duplicate content issues.

Tested Function

parseRobotsTxt() from robots-txt-parser.js
checkInternalSearchPagesBlocked() from robots-txt-parser.js

Issue Information

Issue Number: 166
Issue Code: SEARCH_RESULTS_BLOCKED
Toggle Group: crawlBehaviour

Test Scenarios

Positive Test Cases

Internal search paths are explicitly allowed (Allow: /search) → hasSearchBlocked is false, no issues

Negative Test Cases

Internal search paths are disallowed (Disallow: /search) → SEARCH_RESULTS_BLOCKED issue created with blockedSearchPaths containing /search

Boundary Cases

None

Edge Cases