ShipcheckRobots.txt generator and tester
Build a correct robots.txt in a minute, with presets for WordPress, online stores, SaaS apps, and staging, and a clear choice for AI crawlers. Then test any path against Googlebot, GPTBot, or any other bot.
ChatGPT, Perplexity, and Claude can cite you; model-training crawlers are blocked.
One per line. * matches anything, $ marks the end of a URL.
robots.txt
User-agent: * Allow: / # Opt out of AI model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent User-agent: Bytespider User-agent: Amazonbot Disallow: /
Upload it to the root of your domain so it loads at /robots.txt.
Test a URL path
Allowed: other bots on /
The two-minute robots.txt checklist
- Never ship a staging file with
Disallow: /to production. - Do not block CSS or JavaScript; Google needs them to render your pages.
- List your sitemap with a full URL.
- Decide on AI crawlers on purpose instead of by accident.
Read the full guide: robots.txt for AI crawlers.
Questions
Should I block GPTBot and other AI crawlers?
Block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) if you do not want your content used to train models. Do not block search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) unless you are fine with ChatGPT search, Perplexity, and Claude rarely showing or citing your pages.
Does robots.txt hide a page from Google?
No. It stops crawling, not indexing. A blocked page can still appear in results if other sites link to it. Use a noindex meta tag (and let Google crawl it) or password protection to keep a page out.
Why does a named User-agent group ignore my * rules?
Under RFC 9309 a crawler uses only the most specific group that names it. If you add a GPTBot group, GPTBot ignores everything under User-agent: *. This generator only adds named groups that block, so your * rules keep applying to everyone else.
Where do I put robots.txt?
At the root of each host: https://example.com/robots.txt. A subdomain such as blog.example.com needs its own file.
Related tools
AI crawler checker
See which of 16 AI crawlers your robots.txt allows or blocks, with a copy-paste fix.
Open tool →llms.txt generator
Build an llms.txt file from your sitemap so AI tools that read llms.txt get a clean map of your site.
Open tool →Meta tag & social preview checker
See your Google, X, LinkedIn, and Slack previews and fix titles, descriptions, and OG images.
Open tool →