Is Cloudflare Secretly Blocking AI From Your Site?

If ScoutRival’s Site Audit flagged “Firewall/CDN not blocking AI bots,” it means something at your site’s edge — Cloudflare, a security plugin, or your host’s firewall — is turning AI crawlers away before they ever reach your content. Your robots.txt might say “come on in,” but the bouncer at the door is saying “no.” This is the sneakiest AI-visibility problem there is, because nothing on your actual site looks wrong. This guide shows you how to find it and fix it, on Cloudflare and every other common setup. Most of it is point-and-click, though the deepest host WAF cases can need a developer.

Why this is the most dangerous block of all

Every other crawler problem shows up somewhere you can see it. A firewall block doesn’t. Here’s the chain:

1. An AI bot (say, PerplexityBot) requests your page.
2. Your CDN or WAF inspects the request, decides it’s “a bot,” and returns a 403 Forbidden, a 429 Too Many Requests, or a JavaScript/CAPTCHA challenge.
3. The bot never gets the HTML — so it never reads your beautifully-configured robots.txt, never sees your content, and simply moves on.

To you, the site looks perfect. To your visitors, it works. To Google (which is usually whitelisted), rankings hold. The only place this shows up is in your server logs or a deliberate probe with a bot user-agent. That’s why it can hide for months — and why our audit tests the live response instead of trusting robots.txt.

What changed in 2026: Cloudflare AI Crawl Control

Cloudflare used to offer a single “Block AI bots” switch. In 2026 they replaced it with AI Crawl Control, a more granular system that sorts AI crawlers into three buckets:

• Search — bots that index and cite you in AI answers (this is the one you almost always want allowed).
• Agent — bots that fetch a page live, on demand, to complete a user’s request. This includes ChatGPT-User, Perplexity-User and Claude-User — the on-demand fetchers behind AI answers.
• Training — bots gathering data for future model training (GPTBot, ClaudeBot, and so on). Blocking these is a legitimate licensing choice — see Should You Block GPTBot & ClaudeBot?.

The important 2026 detail: Cloudflare announced that starting September 15, 2026, new domains block Training and Agent bots by default on ad-displaying pages, while Search stays allowed; site owners can opt out beforehand. Because “Agent” covers the live-fetch bots that build a lot of AI answers, a brand-new Cloudflare site can quietly refuse the exact crawlers that assemble responses — without you ever touching a setting. If your domain is new or you didn’t set this deliberately, assume it needs checking.

Step 1 — Fix it in Cloudflare

1. Log into the Cloudflare dashboard and select your domain.
2. In the sidebar, open AI Crawl Control → Crawlers. You’ll see a table of known AI crawlers with an Action column.
3. Under Actions, set the AI search bots to Allow — at minimum OAI-SearchBot, PerplexityBot and Claude-SearchBot, plus their live-fetch counterparts ChatGPT-User, Perplexity-User and Claude-User.
4. Set your Search / Agent posture deliberately: allow both for maximum visibility. Training is the real trade-off — allow it to feed future model memory, or block it to keep content out of training. On a new domain near or after September 15, 2026, explicitly set Agent = Allow so the default block doesn’t catch your answer bots.
5. Now check Security → WAF (and Bot Management, if you’re on that plan) for any custom rule named something like “Block AI Scrapers” that returns a 403. If you find one, change its action to Allow or scope it so it doesn’t catch the search/agent bots.

That combination — AI Crawl Control set to allow, and no WAF rule overriding it — clears the Cloudflare case.

Step 2 — Check your WordPress security plugin

Cloudflare isn’t the only edge that blocks bots. Security plugins do too:

• Wordfence: go to Wordfence → Firewall → Blocking and review any blocked user-agents or “block fake/aggressive bots” rules; whitelist the AI answer bots. Also check any rate-limiting rules that might be throttling them into 429s.
• Sucuri / host-level WAF: these are managed at the network layer, so you’ll usually open a support ticket asking them to allow OAI-SearchBot, PerplexityBot and Claude-SearchBot (and the -User fetchers). They can confirm the change in their logs.

Step 3 — Other hosts and CDNs

If you’re on Fastly, AWS WAF, Akamai, Kinsta’s edge, or another managed platform, the same principle applies: an edge rule is deciding your AI bots are unwanted traffic. The fix is to allow those user-agents in the WAF/bot rules, then confirm the change in server logs. On these platforms it’s often a developer or host task rather than a dashboard toggle — but the ask is simple and specific: “please allow OAI-SearchBot, PerplexityBot and Claude-SearchBot (and their -User agents) to return 200 instead of 403.”

Non-Cloudflare, non-WordPress site owners: the block is almost never in your CMS. Look at whatever sits in front of your origin — the CDN, the WAF, or a host-level bot filter.

Step 4 — Verify with a real bot probe

This is the one criterion you can’t confirm by reading robots.txt. You have to test the live response:

1. Manual check (a developer can run this):

   curl -A "OAI-SearchBot" -I https://yourdomain.com
   

   You want the first line to be HTTP/... 200, not 403 or 429. Repeat with -A "PerplexityBot" and -A "Claude-SearchBot".
2. Cloudflare: the AI Crawl Control → Crawlers table should show your target bots as Allowed.
3. Re-run your ScoutRival Site Audit — the AC3 probe fetches your site with each bot user-agent and confirms a 200. When they all pass, the check turns green.

If a bot still gets a 403 after you’ve allowed it in AI Crawl Control, there’s a second rule overriding it — usually a custom WAF rule (Step 1.5) or a plugin (Step 2). Work outward from the CMS to the edge until the probe returns 200.

Common mistakes to avoid

• Assuming a clean robots.txt means you’re reachable. The firewall never lets the bot get to robots.txt. Always probe the live response with a bot user-agent.
• Ignoring the September 15, 2026 default. New Cloudflare domains block Agent (and Training) bots by default on ad pages. If you launched recently, check AI Crawl Control even if you never touched it.
• Blocking “Agent” without realising it holds the answer bots. ChatGPT-User, Perplexity-User and Claude-User live in the Agent category. Blocking Agent wholesale can cut you out of live AI answers.
• Leaving a “Block AI Scrapers” WAF rule in place. A custom firewall rule can silently override your AI Crawl Control setting. Check Security → WAF for one.
• Confusing rate-limiting with blocking. A 429 (too many requests) is just as fatal as a 403 to an impatient AI fetcher. Whitelist the bots from aggressive rate limits too.
• Fixing Cloudflare but forgetting the host WAF. Some stacks have two layers (CDN and host firewall). If the probe still fails, the second layer is the culprit.

The bottom line

A firewall block is the AI-visibility problem that leaves no fingerprints — your site looks flawless while the bots that build ChatGPT, Perplexity and Claude answers are turned away at the edge. In Cloudflare, open AI Crawl Control → Crawlers and set the AI search and agent bots to Allow; clear any “block AI” WAF rule; whitelist the same bots in Wordfence or your host firewall; then prove it with a live curl probe returning 200.

Because this one hides from every robots.txt check, it’s exactly where an automated probe earns its keep. Run a free Site Audit with ScoutRival — it fetches your site as each AI bot and tells you which ones get a 200 and which get a 403, with a plain-English fix for every block it finds.