ShipcheckRobots.txt for AI crawlers: get cited by ChatGPT, Perplexity, and Claude
The difference between AI search and AI training bots, the exact robots.txt rules to use, and the mistakes that make sites invisible to AI answers.
Short version: allow AI search crawlers if you want to be cited in ChatGPT, Perplexity, and Claude. Block AI training crawlers only if you want to opt out of model training. Check your current setup with the AI crawler checker.
Search bots vs training bots
AI companies run separate crawlers for separate jobs. Search crawlers fetch pages so an assistant can answer a question and link to you as a source. Training crawlers collect text to train future models. They use different user agents, so robots.txt can treat them differently.
That distinction is the whole game. Many sites blocked "AI bots" in 2023 with one broad rule and quietly dropped out of AI search answers, which are now a real source of traffic and signups.
The AI crawlers that matter
| User agent | Company | Used for |
|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Pages fetched during a chat |
| GPTBot | OpenAI | Training |
| Claude-SearchBot | Anthropic | Claude search results |
| Claude-User | Anthropic | Pages fetched during a chat |
| ClaudeBot | Anthropic | Training |
| PerplexityBot | Perplexity | Perplexity answers |
| Googlebot | Google Search and AI Overviews | |
| Google-Extended | Gemini training and grounding (not Search) | |
| Bingbot | Microsoft | Bing and Copilot |
| Applebot-Extended | Apple | Control token: opts Applebot-crawled content out of Apple model training |
| CCBot | Common Crawl | Open dataset used to train many models |
| Meta-ExternalAgent | Meta | Training |
Copy-paste recipes
Be cited in AI answers, opt out of training
The setup most businesses want:
User-agent: * Allow: / # Opt out of AI model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent Disallow: / Sitemap: https://example.com/sitemap.xml
Search crawlers have no group of their own, so they follow your * rules. Training crawlers are blocked by name. Build your own version with the robots.txt generator.
The named-group trap
Under the robots.txt standard (RFC 9309), a crawler follows only the most specific group that names it and ignores the * group completely. Adding a group to "allow" a bot can accidentally open private paths:
User-agent: * Disallow: /admin/ User-agent: OAI-SearchBot Allow: / # OAI-SearchBot now ignores the * group, # so it CAN crawl /admin/. Repeat the rule: Disallow: /admin/
Mistakes that hide you from AI
- Blocking everything with one rule. A list of every "AI" user agent with
Disallow: /includes the search bots too. - CDN and firewall settings. Bot protection (for example Cloudflare's AI crawler blocking) can stop crawlers before they ever read robots.txt. Check your dashboard, not just the file.
- JavaScript-only content. Most AI crawlers do not run JavaScript. If your page is an empty shell until React loads, they see nothing. Use server rendering or static generation for content pages.
- noindex on important pages. Google's AI Overviews and AI Mode respect noindex like the rest of Search; other AI products vary.
- Staging rules on production.
Disallow: /copied from staging blocks every crawler, AI or not.
Beyond robots.txt
- Publish an llms.txt file with a short summary and links to your key pages.
- Add structured data so assistants get your name, product, and prices without guessing.
- State the answers people ask for in plain text: what you do, who it is for, pricing, and how you compare. Assistants quote clear sentences.
Questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and fetches made while a user chats use ChatGPT-User. Block GPTBot to opt out of training while staying visible in ChatGPT answers.
Does Google-Extended affect Google Search or AI Overviews?
No. Google-Extended controls whether your content is used for Gemini models and grounding. Google Search, including AI Overviews, is controlled by Googlebot. Blocking Googlebot stops Google from reading your pages, so they effectively drop out of Search.
Do AI crawlers respect robots.txt?
The major ones from OpenAI, Anthropic, Google, Apple, and Perplexity document their user agents and say their crawlers follow robots.txt. User-triggered fetchers act for a person in real time: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, while Anthropic says Claude-User follows robots.txt. For hard blocking, use your firewall or CDN.
Is llms.txt a standard?
It is a community proposal from 2024, not an official standard, and AI companies have not committed to using it. It is cheap to add and gives assistants a clean summary of your site, so many sites publish one anyway.
How long until changes take effect?
Crawlers re-read robots.txt regularly, typically within a day. Getting cited in AI answers after unblocking can take longer, because your pages need to be re-crawled and indexed.
Tools for this guide
AI crawler checker
See which of 16 AI crawlers your robots.txt allows or blocks, with a copy-paste fix.
Open tool →Robots.txt generator & tester
Generate robots.txt with AI crawler presets and test any path against any bot.
Open tool →llms.txt generator
Build an llms.txt file from your sitemap so AI tools that read llms.txt get a clean map of your site.
Open tool →More guides
The website launch checklist (2026)
48 checks across SEO, sharing, accessibility, AI search, speed, security, and legal. Tick them off as you go and download your progress.
Read · 12 min →European Accessibility Act: a practical guide for small teams
Who the EAA applies to, what it requires, and a step-by-step plan to work toward compliance without an enterprise budget.
Read · 10 min →Core Web Vitals fixes: LCP, CLS, and INP in plain English
What each metric measures, the thresholds Google uses, and the fixes that actually move the numbers on real sites.
Read · 11 min →