Cloudflare, CDN, and WAF settings that block AI crawlers
Why ChatGPT and Perplexity still fail when robots.txt allows them: bot fight modes, AI scrapers rules, and how to verify real crawler access.
Short answer
robots.txt only asks crawlers to behave. A CDN, Cloudflare Bot Fight Mode, “Block AI scrapers,” or WAF rule can still return 403 even when OAI-SearchBot is allowed. Check your edge bot settings for OpenAI, Anthropic, and Perplexity, then re-test with an AI crawler checker and confirm the server HTML is not empty.
Start with the AI crawler checker and AI citation readiness checker. If robots.txt looks fine but the page still fails, read this guide next.
robots.txt vs the edge
robots.txt tells well-behaved crawlers which paths they may request. It does not open your firewall. CDNs and WAFs decide whether the TCP/TLS request reaches your origin at all. A common failure mode: robots.txt allows OAI-SearchBot, while Cloudflare or another edge product still challenges or blocks that user agent.
That is why Shipcheck tools warn that a firewall or CDN bot rule can block crawlers even when robots.txt allows them. Fix both layers or AI search still sees a wall.
Symptoms
- robots.txt shows search bots as allowed, but paste-URL tools get 403 or a captcha HTML page.
- What crawlers see shows challenge text, Cloudflare branding, or almost no real content.
- ChatGPT / Perplexity never cite you despite thin-HTML and robots fixes elsewhere.
Cloudflare checks
- Open Security → Bots (or your plan’s Bot Fight / Super Bot Fight settings). Note whether “definitely automated” traffic is blocked or challenged.
- Review any Block AI Scrapers / AI bot management rules. Decide which bots you want for search citations vs training opt-out, and allow the search crawlers you care about.
- Check WAF custom rules and rate limits that match User-Agent or ASNs for OpenAI, Anthropic, or Perplexity.
- Confirm you are not forcing a JS challenge on all unknown bots for HTML pages you want cited — many AI search crawlers do not complete those challenges.
Align edge policy with your robots.txt for AI crawlers guide: search yes / training no is a common pattern.
Other CDNs and WAFs
- Vercel / Netlify / host WAF: look for bot protection, geo blocks, and IP allowlists that reject datacenter crawlers.
- AWS WAF / Fastly / Akamai: review managed bot rules and custom matches on User-Agent.
- Staging auth: basic auth or VPN walls block AI crawlers the same way they block Google — expected for private staging, fatal for production URLs you want cited.
Verify the fix
- Re-run the AI crawler checker — robots.txt should still allow the search bots you want.
- Fetch the page HTML (or use crawler-view) and confirm you get real content, not a challenge shell.
- Score readiness with the citation readiness checker, then follow how to get cited by AI search for content and mentions.
Allowing a crawler at the edge still does not guarantee a citation — it only removes a hard block before content quality matters.
Questions
robots.txt allows OAI-SearchBot — why am I still blocked?
robots.txt is a request, not a network allowlist. Cloudflare Bot Fight Mode, “Block AI Scrapers,” AWS WAF, or host firewall rules can still return 403 or a challenge page to those user agents.
Does Cloudflare’s AI scrapers block stop ChatGPT search?
If the rule blocks OpenAI’s search crawler (or challenges it into a JS interstitial), ChatGPT search may never see your HTML even when robots.txt allows OAI-SearchBot. Review which AI bots you intend to allow for search vs training.
How do I confirm the edge is the problem?
If robots.txt allows the bot but a fetch of the page returns 403, a challenge HTML page, or almost no content, check CDN/WAF logs and bot settings. Pair that with Shipcheck’s AI crawler checker and crawler-view tools.
Should I allow every AI bot?
No. Many teams allow search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and still block training crawlers (GPTBot, ClaudeBot, Google-Extended). Edge rules should match that policy.
Tools for this guide
AI crawler checker
See which of 16 AI crawlers your robots.txt allows or blocks, with a copy-paste fix.
Open tool →AI Visibility Audit
Can ChatGPT, Claude, and Perplexity read your site? Score, crawler chips, sitemap sample, Master Prompt. Readiness only — not a citation guarantee.
Open tool →Robots.txt generator & tester
Generate robots.txt with AI crawler presets and test any path against any bot.
Open tool →More guides
Keyboard focus order: catch tab traps before launch
What focus order means for WCAG, how to test with the keyboard alone, and how Shipcheck Pro’s overlay shows missing focus styles and unreachable controls.
Read · 8 min →Shipcheck fix prompts for Cursor, Claude, and ChatGPT
How free sample prompts differ from Pro’s every-issue prompts, what to paste into Cursor or Claude, and how to keep AI edits minimal and reviewable.
Read · 7 min →Title tags and meta descriptions: length, examples, and how to write ones that get clicks
How long a title tag and meta description should be in 2026, what Google actually shows, and copy patterns that improve CTR without keyword stuffing.
Read · 9 min →