Skip to content
Rezultra®
Request a callback

Home / Tools / robots.txt Tester

robots.txt tester

Edit robots.txt and see instantly which search engines and AI bots can crawl any URL. Then generate rules that keep you in AI search.

Sitemaps declared: https://www.example.com/sitemap.xml

  • OAI-SearchBotOpenAI · AI search & answers
    No matching rule (* group)
    Allowed
  • GPTBotOpenAI · Model training
    Line 6: Disallow: / (its own group)
    Blocked
  • ChatGPT-UserOpenAI · Fetches on user request
    No matching rule (* group)
    Allowed
  • Claude-SearchBotAnthropic · AI search & answers
    No matching rule (* group)
    Allowed
  • ClaudeBotAnthropic · Model training
    No matching rule (* group)
    Allowed
  • Claude-UserAnthropic · Fetches on user request
    No matching rule (* group)
    Allowed
  • PerplexityBotPerplexity · AI search & answers
    No matching rule (* group)
    Allowed
  • Perplexity-UserPerplexity · Fetches on user request
    No matching rule (* group)
    Allowed
  • GooglebotGoogle · AI search & answers
    No matching rule (* group)
    Allowed
  • Google-ExtendedGoogle · Model training
    No matching rule (* group)
    Allowed
  • BingbotMicrosoft · AI search & answers
    No matching rule (* group)
    Allowed
  • ApplebotApple · AI search & answers
    No matching rule (* group)
    Allowed
  • Applebot-ExtendedApple · Model training
    No matching rule (* group)
    Allowed
  • DuckAssistBotDuckDuckGo · AI search & answers
    No matching rule (* group)
    Allowed
  • CCBotCommon Crawl · Model training
    No matching rule (* group)
    Allowed
  • Meta-ExternalAgentMeta · Model training
    No matching rule (* group)
    Allowed
  • Meta-ExternalFetcherMeta · Fetches on user request
    No matching rule (* group)
    Allowed
  • AmazonbotAmazon · AI search & answers
    No matching rule (* group)
    Allowed
  • BytespiderByteDance · Model training
    No matching rule (* group)
    Allowed
  • AhrefsBotAhrefs · SEO tools
    No matching rule (* group)
    Allowed
  • SemrushBotSemrush · SEO tools
    No matching rule (* group)
    Allowed

Generate AI-friendly rules

Stay visible in AI search while choosing whether models may train on your content.

AI model training

Repeated in every group, because a bot with its own group ignores the * rules.

robots.txt
# Bots named in a group skip the * group, so each group repeats your private paths.

# AI search engines: let them read and cite your pages
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckAssistBot
User-agent: Amazonbot
Allow: /
Disallow: /admin/
Disallow: /cart

# Assistants fetching a page because a user asked
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
Allow: /
Disallow: /admin/
Disallow: /cart

User-agent: *
Disallow: /admin/
Disallow: /cart

Sitemap: https://www.example.com/sitemap.xml

Test rules before you ship them

A single misplaced Disallow: / can remove a site from Google and every AI assistant. This tester runs entirely in your browser using the same matching rules crawlers use: wildcards (*), end anchors ($), longest-match-wins and Allow-wins-ties.

A sensible default for 2026

Most businesses want to be found in AI search. That means allowing the search crawlers of ChatGPT, Claude, Perplexity, Google and Bing. Whether AI companies may train on your content is a separate decision, and the generator lets you make it without affecting your visibility.

Free tools

More from the toolkit

Website SEO Audit

Crawls up to 10 pages of your site and scores SEO, AI readiness, speed and structured data, with every problem ranked by impact.

10 pages free

Social Media Benchmark

Shows your YouTube channel’s public numbers exactly as YouTube reports them, next to channels you choose, and checks that your website and channel are connected. Type in your Instagram numbers to score your engagement and posting. Facebook coming soon.

YouTube · Instagram (your numbers) · Facebook soon

AI Crawler Checker

Tests robots.txt against 19 search and AI crawlers, sends a live request as 12 of them, and separates AI search from AI training.

19 crawlers

Core Web Vitals

LCP, INP and CLS from real Chrome users with 25-week trends, plus a Lighthouse test that lists the biggest fixes.

LCP · INP · CLS

Schema Validator

Validates structured data on any page or pasted code against Google’s rich-result requirements, in plain English.

Rich results

SERP & Social Preview

Pixel-accurate Google snippets for mobile and desktop and social share cards. Edit live and see where text is cut off.

Pixel accurate

llms.txt Generator

Builds an llms.txt file from your site in seconds, with an honest note on what the evidence says it does.

Courtesy file
Rezultra

Want us to fix it for you?

The tools show you what’s wrong. Our team fixes it and measures the result in revenue, not rankings.

Request a callback