Skip to Content
Geo and AI Engine SignalsIssue 110

No llms.txt published

What Is This Issue

llms.txt is a plain-text Markdown file placed at the root of a website (e.g., https://example.com/llms.txt). It helps AI language models discover and understand a site’s key content in a structured, LLM-friendly format. This issue checks whether your site has an llms.txt file, whether it’s properly formatted, and accessible.

A passing implementation requires:

  • An llms.txt file exists at your root domain
  • The file returns HTTP 200 OK with Content-Type: text/plain
  • The file follows proper Markdown structure (H1 heading, blockquote description, section headings)
  • All listed URLs in the file return 200 OK status
  • The file size is under the recommended limit (50-100KB)
  • A discovery tag is present in your homepage <head>

Example: A properly configured llms.txt file at https://example.com/llms.txt with a site title, description, section headings, and links to key pages.

Why Is This Important

The llms.txt file is important for SEO because it:

  • Improves AI Discoverability: Helps AI language models find and understand your most important content
  • Enhances AI Search/AEO: Increases chances of your content being referenced in AI-generated answers
  • Provides Structured Context: Gives LLMs a curated index of your site instead of parsing entire HTML pages
  • Future-Proofs SEO: As AI search grows, having an llms.txt file positions your site for AI-driven discovery

While still a proposed standard (not yet officially adopted by all AI models), llms.txt is increasingly used by technical sites and AI platforms. Implementing it early can give you an advantage in AI search visibility.

Resolving this issue improves your overall SEO health score by ensuring your site is optimized for the growing AI search ecosystem.

How XeoPix Detects This

XeoPix performs llms.txt validation through the following logical steps:

  1. File Discovery: XeoPix attempts to fetch {site_root}/llms.txt and records the HTTP status code.

  2. Format Validation: If the file exists (returns 200 OK), XeoPix checks:

    • The Content-Type header is text/plain (not HTML or JSON)
    • The file starts with an H1 heading (#)
    • The file has a blockquote description (>)
    • The file has at least one section heading (##)

  3. Link Validation: XeoPix extracts all URLs listed in the Markdown file and sends HEAD requests to verify they return 200 OK.

  4. Size Check: XeoPix checks the file size to ensure it doesn’t exceed recommended limits.

  5. Discovery Tag Check: XeoPix parses your homepage <head> to verify the <link rel="llms"> tag is present.

  6. Issue Identification: XeoPix raises issues when:

    • llms.txt file is missing (STANDARD)
    • Server returns 5xx error (IMPORTANT)
    • File is not properly formatted as Markdown (IMPORTANT)
    • Listed URLs return non-200 status (IMPORTANT)
    • File exceeds size limits (STANDARD)
    • Discovery tag is missing from homepage (STANDARD)

How To Fix

  1. Create the file: Create a plain text file named exactly llms.txt at your domain root (https://example.com/llms.txt).

  2. Structure with Markdown: Format the file using proper Markdown:

    • Start with an H1 heading: # Your Site Name
    • Add a blockquote description: > Brief description of your site
    • Add section headings: ## Documentation, ## Products, etc.
    • Add links: - [Page Title](https://example.com/page): Description

  3. Serve with correct headers: Ensure the file returns:

    • HTTP 200 OK status
    • Content-Type: text/plain header

  4. Validate links: Ensure all URLs listed in the file return 200 OK when accessed.

  5. Add discovery tag: Include this tag in your homepage <head>:

    <link type="text/plain" rel="llms" href="/llms.txt" />

  6. Keep file size reasonable: Aim for under 50KB to ensure fast processing by LLMs.

  7. Test the file: Validate that:

    • The file is accessible at the root URL
    • The Markdown is properly formatted
    • All listed links work
    • The discovery tag is present in your homepage

What We Store

Storage Level

Site Level

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

FieldTypeDescription
robotsTxtDataJson?Robots.txt content and directives in JSON format

Detection Dependencies

  • robots.txt

Examples

Example 1: Missing llms.txt file

Problematic state (fails):

  • No llms.txt file exists at https://example.com/llms.txt
  • XeoPix returns a 404 error when attempting to fetch the file
  • Issue raised: “llms.txt file is missing”

Corrected state (passes):

  • Create llms.txt file at the root domain
  • File returns HTTP 200 OK
  • Issue resolved

Example 2: Improperly formatted llms.txt

Problematic state (fails):

My Website
This is my website description.
Some links:
 
- https://example.com/page1
- https://example.com/page2

Issues:

  • Missing H1 heading (#)
  • Missing blockquote description (>)
  • No section headings (##)
  • URLs not formatted as Markdown links

Corrected state (passes):

# My Website
 
> This is my website description.
 
## Documentation
 
- [Getting Started](https://example.com/docs): Learn how to use our product
- [API Reference](https://example.com/api): Complete API documentation
 
## Products
 
- [Product A](https://example.com/product-a): Our flagship product

Example 3: Missing discovery tag

Problematic state (fails):

  • llms.txt file exists and is properly formatted
  • Homepage HTML does not include the discovery tag
  • Issue raised: “Discovery tag is missing from homepage”

Corrected state (passes): Add to homepage <head>:

<link type="text/plain" rel="llms" href="/llms.txt" />

XeoPix detects the tag and the issue is resolved.

Unit Test

Test File

xeopix-crawling-v2/__tests__/seo-audit-checks/geoAiEngineSignals/issue-186-llms-txt-directive.test.js

Purpose

Validates the llms.txt directive detection in robots.txt — ensuring the system correctly identifies whether a site declares an llms.txt file via the llms.txt: directive in its robots.txt, and persists the corresponding audit issue when the directive is missing.

Tested Function

runGeoAiEngineSignals from toggleGroups/geoAiEngineSignals.js

Issue Information

  • Issue Number: 186
  • Issue Code: llms_txt_file
  • Toggle Group: geoAiEngineSignals

Test Scenarios

Positive Test Cases

  1. llms.txt directive present and accessible
    • robots.txt contains: User-agent: *\nAllow: /\nllms.txt: https://example.com/llms.txt
    • llms.txt file returns HTTP 200 with content
    • Expects hasLlmsTxt: true, llmsTxtCount: 1, llmsTxtUrls: ['https://example.com/llms.txt']
    • Expects no detected issues — the detectedIssues array is empty
    • Expects no POST to audit-issues/bulk

Negative Test Cases

  1. llms.txt directive missing from robots.txt
    • robots.txt contains: User-agent: *\nAllow: / (no llms.txt: directive)
    • llms.txt file returns HTTP 404
    • Expects hasLlmsTxt: false, llmsTxtCount: 0, llmsTxtUrls: []
    • Expects a detected issue with issueCode equal to IssueCode.LLMS_TXT_FILE
    • Expects a POST to audit-issues/bulk containing the issue in the items array

Boundary Cases

None explicitly tested.

Edge Cases

None explicitly tested.

Expected Outcome

Pass

The issue should be reported (i.e., detectedIssues contains LLMS_TXT_FILE) when:

  • robots.txt exists but does not contain an llms.txt: directive.

Fail

The issue should not be reported (i.e., detectedIssues is empty) when:

  • robots.txt contains an llms.txt: directive pointing to a valid, accessible llms.txt file.

Validation

The unit test verifies:

  • Correct issue detection: When the llms.txt: directive is absent from robots.txt, the function returns hasLlmsTxt: false and includes LLMS_TXT_FILE in detectedIssues.
  • No issue detection when expected: When the llms.txt: directive is present and the file is accessible, the function returns hasLlmsTxt: true with no detected issues.
  • Payload validation: The llmsTxt result object contains the correct shape (hasLlmsTxt, llmsTxtCount, llmsTxtUrls, detectedIssues).
  • Persistence validation: When an issue is detected, a POST request is made to audit-issues/bulk with the correct issueCode in the payload.
  • No unnecessary persistence: When no issue is detected, no POST request is made to audit-issues/bulk.
  • toggleGroups/geoAiEngineSignals.js — Contains the runGeoAiEngineSignals function under test.
  • robots-txt-parser.js — Contains the checkLlmsTxtDirective function that parses the llms.txt: directive from raw robots.txt text.
  • issueCodes.js — Defines the IssueCode.LLMS_TXT_FILE constant.

Coverage Summary

  • Covers both the positive and negative paths for llms.txt: directive detection in robots.txt.
  • Validates the complete flow: fetching robots.txt → parsing the directive → returning the result → persisting the issue when applicable.
  • Validates that no unnecessary API calls are made when the directive is properly configured.
  • Does not cover llms-full.txt (issue #199) detection, format validation, link validation, size checks, or discovery tag checks — these are tested separately or not yet covered.

References

Last updated on
Issue 109Issue 111