No PerplexityBot rule in robots.txt

What Is This Issue

This issue checks whether a website has an explicit rule in robots.txt for PerplexityBot, the web crawler used by Perplexity AI to collect data for answering user queries. Allowing PerplexityBot enables site content to surface in Perplexity AI answers.

A passing implementation means:

• The robots.txt file contains an explicit rule for PerplexityBot (either Allow or Disallow)
• The rule is clearly defined and intentional
• If allowed, the site’s content may be cited in Perplexity AI responses

Example (Allow):

User-agent: PerplexityBot
Allow: /

Example (Disallow):

User-agent: PerplexityBot
Disallow: /

Why Is This Important

AI Search Visibility: Perplexity AI is an emerging AI search engine that answers questions by citing web sources. Allowing PerplexityBot means your content may be referenced in Perplexity answers, driving referral traffic.

Content Attribution: Perplexity AI typically provides citations and links back to source content, offering a potential new traffic source different from traditional search engines.

Early Adoption Advantage: As AI search engines gain market share, being indexed early may provide competitive advantages.

Content Protection: If you don’t want your content to be used in Perplexity AI responses, you should explicitly disallow PerplexityBot.

Control Over AI Training: Unlike some AI bots that use data for model training, PerplexityBot primarily uses crawled content for real-time answer generation with attribution.

SEO Health Score: Having explicit AI bot rules demonstrates awareness of emerging AI search technologies and improves the technical SEO score.

How XeoPix Detects This

XeoPix performs the following checks:

1. Fetches robots.txt - The crawler requests the /robots.txt file from the domain being audited.

2. Parses the content - XeoPix reads through the robots.txt file and identifies all User-agent: blocks.

3. Looks for PerplexityBot - The system searches for a User-agent: line that matches PerplexityBot (case-sensitive).

4. Validates the rule - If a PerplexityBot user-agent block is found, XeoPix checks that it contains either:

   • Disallow: directive (with any value)
   • Allow: directive (with any value)

5. Issue Identification: XeoPix raises issues when:

   • No explicit rule for PerplexityBot is found (SUGGESTION)
   • The rule syntax is invalid (WARNING)

How To Fix

1. Decide your policy - Determine whether you want Perplexity AI to crawl and cite your content:

   • Allow: Your content may be cited in Perplexity AI answers with attribution
   • Disallow: Your content won’t be crawled or cited by Perplexity AI

2. Add the rule to robots.txt - Add the following to your robots.txt file:

   User-agent: PerplexityBot
   Disallow: /

   Or to allow:

   User-agent: PerplexityBot
   Allow: /

3. Place it correctly - The rule can be placed anywhere in robots.txt, but many prefer to group AI bot rules together.

4. Test the rule - Verify the syntax is correct using Google Search Console’s robots.txt tester or by checking the file directly.

5. Consider partial rules - You can disallow specific sections while allowing others:

   User-agent: PerplexityBot
   Disallow: /private/
   Allow: /

What We Store

Storage Level

Site Level — This issue is evaluated at the site/domain level.

Database Table / Prisma Model

SiteCrawlBehaviourData

Stored Fields

Detection Dependencies

• The following data sources are required to evaluate this issue:
• robots.txt — The crawler fetches and parses the robots.txt file to check for allow/disallow directives
• HTTP Response — The crawler checks for robots.txt at https://domain.com/robots.txt 

Examples

Example 1: Allowing PerplexityBot

Problematic State (Fails): No explicit rule for PerplexityBot. You want your content to be cited in Perplexity AI answers.

Corrected State (Passes): Add explicit allow rule:

User-agent: PerplexityBot
Allow: /

Example 2: Disallowing PerplexityBot

Problematic State (Fails): No explicit rule for PerplexityBot. You don’t want your content used in Perplexity AI.

Corrected State (Passes): Add explicit disallow rule:

User-agent: PerplexityBot
Disallow: /

Example 3: Grouping AI Bot Rules

Problematic State (Fails): AI bot rules are scattered throughout robots.txt.

Corrected State (Passes): Group AI bot rules together:

# Regular crawlers
User-agent: Googlebot
Allow: /

# AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

References

• PerplexityBot Documentation  — Perplexity AI
• Robots.txt Introduction  — Google Search Central