Three kinds of AI crawler
Every major AI company now runs separate bots for separate jobs, and robots.txt controls each one on its own:
- Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) build the index that AI answers are drawn from. Block them and that assistant can’t cite you.
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent) collect text to train future models. Blocking them is a legitimate business choice and doesn’t affect AI search.
- User fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page when someone asks an assistant to read it.
Why we test your live server too
robots.txt is only half the story. Firewalls, CDNs and security plugins often block bots before they ever read robots.txt. We send one clearly labelled request as each of 12 crawlers (OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, CCBot, Meta-ExternalAgent), using its published user-agent, and report what came back: your page, an error, or a bot challenge. The other 7 are checked against robots.txt only.
Want to edit your rules? Use the robots.txt tester and generator.