Skip to Content
Crawl BehaviourIssue 25

Broken internal links and pages

What Is This Issue

Broken internal URLs are links within your website that point to pages or resources that cannot be loaded. This issue checks for broken internal links (pages, images, scripts, stylesheets, videos, and other same-site resources) that return error status codes or cannot be reached due to network failures.

A passing implementation requires:

  • All internal links return HTTP 200 OK status codes
  • Images, CSS, and JavaScript files are accessible and return proper content types
  • No DNS failures, timeouts, or connection errors for internal resources
  • Proper server routing for all internal page paths

Example: A link on your homepage pointing to /about-us that returns a 404 Not Found error because the page was moved or deleted.

Why Is This Important

Broken internal URLs are critical for SEO because they:

  • Harm Crawlability: Search engines cannot crawl and index pages that return errors
  • Waste Crawl Budget: Crawlers spend time trying to access broken pages instead of valid content
  • Hurt Rankings: Broken links signal poor site maintenance and can negatively impact rankings
  • Degrade User Experience: Users encounter 404 errors or missing images, reducing trust and engagement
  • Affect Indexability: Pages with broken internal links may not be discovered or indexed properly

Broken internal URLs directly impact your SEO health score by preventing search engines from properly crawling and indexing your content. They also create a poor user experience that can increase bounce rates and reduce conversions.

How XeoPix Detects This

XeoPix performs comprehensive broken link detection through the following logical steps:

  1. Link Extraction: XeoPix crawls your pages and extracts all internal links from:

    • <a href> attributes for page links
    • <img src> and <img srcset> for images
    • <link href> for stylesheets
    • <script src> for JavaScript files
    • Other resource references (videos, audio, etc.)

  2. URL Normalization: XeoPix converts all relative URLs to absolute URLs and removes fragment identifiers (the part after #).

  3. Link Validation: XeoPix sends HTTP requests to each internal URL and records:

    • HTTP status codes (200 OK, 404 Not Found, 500 Server Error, etc.)
    • Network errors (DNS failures, timeouts, connection refused)
    • Content types for assets (ensuring images return image/* content types)

  4. Redirect Following: XeoPix follows redirects (301, 302, etc.) up to a reasonable limit to check the final destination.

  5. Issue Identification: XeoPix raises issues when:

    • Internal links return 4xx or 5xx status codes (CRITICAL)
    • Network errors prevent accessing internal resources (CRITICAL)
    • Assets return wrong content types or empty responses (WARNING)
    • Single-page apps mask broken routes with catch-all responses (SUGGESTION)

Note: XeoPix validates links based on raw HTML responses and does not execute JavaScript. Links generated only by JavaScript cannot be detected or validated.

How To Fix

  1. Fix broken links: Update or correct the href or src attributes to point to the correct URL.

  2. Restore missing pages: If a page was accidentally deleted, restore it or redirect to a relevant alternative.

  3. Re-upload missing assets: If images, CSS, or JavaScript files are missing, re-upload them to the correct path.

  4. Fix server routing: Ensure your server is properly configured to handle all valid URL paths.

  5. Check for typos: Verify that internal links don’t contain typos in the URL path.

  6. Update CMS links: If using a CMS, ensure internal link references are updated when pages are moved or renamed.

  7. Test after fixes: After making changes, re-crawl your site to confirm all internal links are working.

What We Store

Storage Level

Page Level — This issue is evaluated for each individual page and its internal links.

Database Table / Prisma Model

PageInternalLink

Stored Fields

FieldTypeDescription
urlIdStringThe ID of the page being analyzed
targetUrlStringThe internal URL that is linked to
statusCodeInt?HTTP status code of the internal link (if crawled)
urlUrlThe source URL of the page containing the link

Detection Dependencies

  • The following data sources are required to evaluate this issue:
  • HTML Document — The crawler extracts all internal links from the page
  • HTTP Response — The crawler may optionally follow links to check their status
  • Internal Links — All same-domain links are identified and analyzed

Examples

Example 1: Broken Internal Page Link

Problematic State (Fails): A link on the homepage points to a deleted page:

<a href="/old-about-page">About Us</a>

The page /old-about-page returns a 404 Not Found error.

Corrected State (Passes): Update the link to point to the correct page:

<a href="/about">About Us</a>

The page /about returns 200 OK.

Example 2: Missing Image Asset

Problematic State (Fails): An image file has been deleted or moved:

<img src="/images/logo.png" alt="Company Logo" />

The image /images/logo.png returns a 404 error.

Corrected State (Passes): Restore the image or update the path:

<img src="/assets/images/logo.png" alt="Company Logo" />

The image /assets/images/logo.png returns 200 OK with correct content type.

Example 3: Broken JavaScript File

Problematic State (Fails): A JavaScript file returns a 500 server error:

<script src="/js/main.js"></script>

The file /js/main.js returns a 500 Internal Server Error.

Corrected State (Passes): Fix the server issue or restore the file:

<script src="/js/main.js"></script>

The file /js/main.js returns 200 OK with application/javascript content type.

Unit Test

Test File

xeopix-crawling-v2/__tests__/seo-audit-checks/crawlBehaviour/issue-9-no-broken-internal.test.js

Purpose

This unit test validates that the runCrawlBehaviour() function detects internal URLs that were discovered during crawling but were not successfully visited, reporting them as broken internal links.

Tested Function

runCrawlBehaviour() from toggleGroups/crawlBehaviour.js

Issue Information

  • Issue Number: 9
  • Issue Code: NO_BROKEN_INTERNAL
  • Toggle Group: crawlBehaviour

Test Scenarios

Positive Test Cases

  • All discovered internal URLs were successfully visited → no broken internal issues reported

Negative Test Cases

  • An internal URL was discovered but not visited → NO_BROKEN_INTERNAL issue created with the URL and a message containing 'Broken internal page detected'

Boundary Cases

None

Edge Cases

None

Expected Outcome

Pass

A NO_BROKEN_INTERNAL issue is not reported when every discovered internal URL has a corresponding entry in the visited URLs map.

Fail

A NO_BROKEN_INTERNAL issue is reported when a URL in internalUrls is missing from the visitedUrls map, indicating it was discovered but not successfully crawled.

Validation

  • Verifies that matching internalUrls against visitedUrls correctly identifies unvisited URLs
  • Validates issue code, URL, and descriptive message in the detected issue payload
  • xeopix-crawling-v2/toggleGroups/crawlBehaviour.js
  • xeopix-crawling-v2/issueCodes.js

Coverage Summary

  • 2 test cases (one positive, one negative)
  • Tests the core logic of comparing discovered URLs against visited URLs
  • Validates payload structure (issueCode, details.url, details.message)

References

Last updated on
Issue 24Issue 26