ShipcheckFree tools / AI crawler checker
Can ChatGPT, Perplexity, and Claude read your site?
Paste a URL. We read your robots.txt and tell you which AI crawlers are allowed or blocked on that page, whether you publish llms.txt, and how much content is in the server HTML.
Search crawlers vs training crawlers
AI companies run two kinds of bots. Search crawlers such as OAI-SearchBot, PerplexityBot, and Claude-SearchBot fetch pages so the assistant can answer questions and cite you. Training crawlers such as GPTBot, ClaudeBot, and Google-Extended collect content to train models.
Blocking a training crawler does not remove you from AI answers. Blocking a search crawler does: OpenAI, for example, says sites that block OAI-SearchBot are not shown in ChatGPT search answers, apart from plain navigational links. Many sites block everything with one broad rule without noticing.
Why server HTML matters
Most AI crawlers do not run JavaScript. If your content only appears after client-side rendering, they see an empty shell. Server-side rendering or static generation fixes this.