Most WordPress site owners assume Cloudflare is just quietly doing its job in the background. That is usually true for speed and basic protection, but there is a catch many people miss: some Cloudflare settings can also block AI crawlers from reading your content. That matters because tools like ChatGPT, Perplexity, and Google AI Overviews do not magically know your site exists. They depend on crawlers to discover, read, and index what you publish.
If those bots hit a wall, your articles may never get cited in AI answers, summaries, or chatbot responses. In other words, your content can be live, useful, and well written, yet still invisible in the part of search that is growing fastest right now. In this guide, I will walk you through what AI crawlers are, why Cloudflare may be blocking them, and exactly what to change so you can protect your site without disappearing from AI-powered search.
What Are AI Crawlers (And Why Should You Care)?
AI crawlers are automated bots used by companies like OpenAI, Google, Anthropic, and Perplexity to read public web pages. You may see names like GPTBot, GoogleOther-Extended, ClaudeBot, and PerplexityBot. Their job is simple: discover content, understand it, and feed it into AI systems that generate answers, overviews, summaries, and citations.
That is why this matters for WordPress site owners. If your site is blocked, challenged, or served a CAPTCHA these bots cannot pass, your content becomes much harder for AI systems to use. Traditional SEO still matters, but AI-powered search is becoming its own traffic source. Every month, more users are asking AI tools for recommendations, explanations, and source links. If your site is not crawlable, you are leaving that visibility on the table.
Why Cloudflare Is Blocking Them
Cloudflare is excellent at stopping bad traffic, but some of its bot protection features are broad enough to catch legitimate AI crawlers too. Bot Fight Mode, aggressive security settings, and custom firewall rules can all treat these crawlers like suspicious traffic. When that happens, the bot may receive a challenge page, a CAPTCHA, or a straight block.
Cloudflare also introduced an AI Scrapers and Crawlers toggle in the dashboard under Security > Bots. If that option is turned on, Cloudflare will block AI crawlers by default. That can be useful if you want to reduce automated scraping, but many site owners switch it on without realizing they are also making their content harder to find in AI search results. For a WordPress site that depends on discovery, that tradeoff can be expensive.
How to Check if Your Site Is Blocking AI Crawlers (Step-by-Step)
Here is the practical checkup I would do on any WordPress site running behind Cloudflare. It only takes a few minutes, and it tells you whether your setup is helping or hurting AI visibility. If you want a related refresher on security tools, I also recommend reading my best WordPress security plugins guide after this.
- Log into the Cloudflare dashboard and open your site. Make sure you are checking the right zone, especially if you manage more than one domain.
- Go to Security > Bots. Look for Block AI Scrapers and Crawlers. If it is on, Cloudflare is blocking AI bots broadly, which includes the crawlers you probably want to allow.
- Check Security Level under Security > Settings. If the level is set to High or I'm Under Attack, Cloudflare may challenge legitimate crawlers more aggressively than you expect.
- Review Firewall Rules and WAF rules. Search for rules that target known bot user-agents or patterns that could accidentally catch GPTBot, ClaudeBot, GoogleOther-Extended, or PerplexityBot.
- Test what is actually happening. Use a robots.txt checker, inspect Cloudflare Analytics, and look at Security Events to see whether requests from crawlers are being blocked, challenged, or passed.
If you find blocks in any of those spots, you have a very good chance of improving AI crawlability without weakening your site overall.
The Right Cloudflare Settings for WordPress Site Owners
Allow Good AI Crawlers, Block Bad Ones
The balanced approach is not to open the floodgates. It is to stop using a blanket block and replace it with smarter rules. I recommend turning off the global Block AI Scrapers toggle and then using Cloudflare WAF custom rules to allow known, reputable crawlers while still blocking unknown or aggressive scrapers.
In practice, that means being specific. You can allow verified user-agents such as GPTBot, ClaudeBot, GoogleOther-Extended, and PerplexityBot, then keep stronger rules in place for suspicious traffic, rate limits, and obvious abuse. That way, your site stays protected, but your best content can still be discovered and cited by the services people are actually using.
Update Your robots.txt Too
Cloudflare is not the only place where crawler access gets broken. Your robots.txt file can also block crawlers, and I have seen many WordPress sites carry old rules for years without anyone noticing. If you set User-agent: * with a blanket Disallow: /, you may be telling every crawler to stay out, including the ones you wanted to welcome.
Open your robots.txt file and check whether it reflects your current goals. If you want AI visibility, make sure you are not accidentally blocking the crawlers that can help you earn it. This is also a good time to review other AI tools for your WordPress site so your content and automation strategy work together instead of fighting each other.
Consider Using Kinsta for Built-In Cloudflare Integration
If you are on Kinsta, you get a simpler path. I use Kinsta and Elementor personally, and one of the things I appreciate is that their Cloudflare integration is pre-configured for WordPress. That means less manual fiddling with edge rules, bot filters, and caching settings, which is exactly where a lot of site owners get tripped up.
Kinsta handles edge caching and bot filtering at the infrastructure level, so you are not starting from scratch in the Cloudflare dashboard. If you want a managed setup that reduces the chances of accidentally blocking AI crawlers while still keeping performance and security strong, that built-in integration is a real advantage.
Common Mistakes WordPress Site Owners Make with Cloudflare
- Turning on Block AI Scrapers and forgetting about it after the initial setup.
- Setting Security Level to I'm Under Attack permanently, which can cause legitimate crawlers to get challenged too often.
- Writing
robots.txtrules that block all bots, such asUser-agent: *followed byDisallow: /. - Not monitoring Cloudflare Security Events, so they never see what is being blocked in the first place.
- Ignoring the issue because traffic looks fine today, even though AI search is additive and citations are being left behind.
Check out our Kinsta vs WP Engine comparison — see which managed host comes with built-in Cloudflare integration so you spend less time on bot settings and more time growing your site.
It does not usually hurt classic Google rankings directly, but it can reduce your visibility in AI answers, summaries, and citations, which is becoming a separate traffic source.
No, a better approach is to allow known good crawlers and block suspicious or abusive traffic with more specific WAF rules.
Start in Security > Bots, then review Security Level, Firewall Rules, and Security Events to see whether requests are being blocked or challenged.
No, robots.txt is a signal, not a complete enforcement layer. You also need to check Cloudflare and any WordPress security rules that sit in front of your site.
Yes, Kinsta's built-in Cloudflare integration reduces manual setup and helps keep performance, security, and crawlability aligned for WordPress sites.
Final Thoughts
AI-powered search is the new SEO frontier, and it is moving quickly. If your Cloudflare settings are blocking AI crawlers, you are not just losing a little bit of traffic, you are losing citations, authority, and discovery inside ChatGPT, Perplexity, and Google AI Overviews. The good news is that this is usually fixable fast. In many cases, ten minutes in your Cloudflare dashboard is enough to change your site from hidden to visible. If you want to go deeper on how AI security works, that is a smart next read too. Keep your site protected, but do not let overzealous bot rules shut the door on the next wave of search.
For most sites, the fix is not dramatic. You do not need to dismantle Cloudflare, turn off security, or accept more abuse. You just need to separate good automation that helps people find your work from bad automation that wastes your resources. Once you make that distinction, the Cloudflare dashboard becomes easier to manage, and your WordPress content has a much better chance of showing up where readers are now looking.
If you are unsure where to start, begin with the Bots page, then review Security Events after a day or two. That feedback loop tells you whether your changes are helping legitimate crawlers get through while keeping junk out. In practice, that is the difference between guessing and running a site with a deliberate AI search strategy.
About The Author
This happened to one of my client sites last week. Your solution saved us from losing traffic. Thanks!
Glad it helped. On Cloudflare Blocking AI Crawlers WordPress, the big win is usually getting the first pass right and then refining from there. That tends to save a lot of time later.
Hi, I just checked my Cloudflare settings and the ‘Block AI Scrapers and Crawlers’ toggle was on. I turned it off but now I am worried I will get more bad bot traffic. Is there a way to allow the good AI crawlers but still block the sketchy ones?
Great question, Rachel. Yes, that is exactly the right approach. Instead of the blanket block toggle, go into Cloudflare WAF and create custom rules that allow known good AI crawler user-agents (like GPTBot, ClaudeBot, and GoogleOther-Extended) while blocking anything else that looks suspicious. This gives you the best of both worlds. Good instinct to turn off the blanket block first.
Clear steps to protect against Cloudflare’s AI block. Implementing this today.
Thanks for reading, Yuki Yamamoto. On Cloudflare Blocking AI Crawlers WordPress, I would start with the basics and then tighten things up once you see how it behaves in practice. If you are already testing it, the main thing is to keep the setup simple enough to debug quickly.
Did not know Cloudflare was silently blocking AI crawlers. Checked my site and sure enough, the toggle was on. Fixed in 2 minutes. Solid post.
Glad we could help prevent the impact! Spread the word to other site owners. Cloudflare’s announcements are moving fast and not everyone sees them in time.