Should You Block AI Scrapers from Your WordPress Site? (Pros, Cons, and How to Decide)
Every day, AI companies crawl your WordPress site. GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot… they show up quietly in your server logs, reading your blog posts, studying your guides, and in many cases absorbing your hard work into their training data or retrieval systems without so much as a thank you.
So the question is: should you block them?
Some WordPress site owners are slamming the door shut on every AI crawler they can find. Others are rolling out the welcome mat, hoping AI search visibility will become the next organic traffic channel. Most people I talk to, though, are somewhere in the middle. They just do not know what the right call is. They pick a side based on a gut feeling, not data.
This guide will fix that. I will walk you through what AI scrapers actually do, the real pros and cons on both sides, a simple decision framework you can use today, and the exact steps to block or unblock them depending on where you land. By the end, you will know exactly what to do with your site.
What AI Scrapers Are Actually Doing on Your Site
Before you make a decision about blocking, it helps to understand who is showing up and why. AI scrapers are not one monolithic thing. They fall into two broad buckets, and the difference matters.
Training crawlers collect web content to train large language models. GPTBot (OpenAI), ClaudeBot (Anthropic), Amazonbot, Applebot, and Meta’s various crawlers all fall into this category. They visit your site, read everything, and feed it into the next model update. You get zero traffic from these visits. Zero attribution. Your content becomes a tiny drop in a trillion-word dataset. Nobody sees your name attached to it.
Retrieval crawlers are different. These power real-time answers. When someone asks Perplexity AI a question and your blog post is used as a source, PerplexityBot is what made that possible. It visits sites, indexes them, and when relevant, the AI cites your content and links back to you. Google-Extended fills a similar role for Google’s AI products. These crawlers can actually send referral traffic your way.
The key thing to know: most of these crawlers respect robots.txt. If you tell GPTBot to stay out, it generally does. The enforcement is voluntary, and that is a whole other conversation, but as a practical matter, adding a few lines to your robots.txt file stops most AI crawlers from touching your site.
The Case for Letting AI Crawlers In
There is a real argument for keeping the door open, and it is getting stronger as AI search grows.
The biggest reason is visibility. AI-powered search tools (ChatGPT Search, Perplexity, Google AI Overviews, the new Claude Search) now handle billions of queries. If your content is blocked from their crawlers, you are invisible to every single one of those users. That slice of the search pie is growing. Ignoring it today is like ignoring Google in 2005.
Perplexity, in particular, links back to sources. I have seen referral traffic from Perplexity citations show up in my analytics. It is not enormous yet, but it is real. Getting cited as an authority source in AI answers builds brand recognition over time. Someone reads your analysis twice in AI answers, and the third time they search for you directly.
There is also the long game of training data inclusion. If your expert content is part of a model’s knowledge base, and that model is used by millions, your perspective becomes part of the baseline understanding. That is abstract and hard to measure, but for authority sites building a brand, it is not nothing.
Finally, most AI crawlers are polite. They do not hammer your site the way malicious scrapers do. If your hosting can handle regular traffic, GPTBot is unlikely to be the thing that brings you down.
The Case for Blocking Them
The counterargument is equally valid, and for some site owners it wins decisively.
The most immediate concern is server load. While individual AI crawlers tend to be polite, the combined volume is growing fast. Cloudflare’s data shows AI crawler traffic is one of the fastest-growing categories of bot traffic on the web. On cheap shared hosting, even well-behaved crawlers can spike your CPU usage and slow things down for real visitors. If you are on solid managed hosting, like the kind that handles AI-heavy WordPress sites without breaking a sweat, this is less of a worry. But if your hosting is already strained, every bot counts.
Then there is the content competition problem. When an AI summarizes your article in a chat response, the user gets the answer without ever visiting your site. If your business model depends on page views (display ads, affiliate clicks, product sales), that is a direct revenue hit. Your content is generating value for OpenAI or Perplexity, not for you.
There is also the derivative content risk. Your original research, case studies, or unique frameworks get absorbed into an AI and then regurgitated without context or attribution. Someone asks the AI to “write a guide about X,” and the AI spits out a heavily paraphrased version of your work. That is your intellectual property being laundered through a model.
Privacy is another angle. If you publish anything personal, proprietary, or client-related, having AI models store and potentially reproduce that data is a real liability. It is not just about your blog posts. It is about everything on your domain.
And an honest truth: the vast majority of AI-generated answers will never include a link back to your site. GPTBot and ClaudeBot visit your pages, learn from them, and move on. You receive nothing in return.
How to Decide: A Simple Framework
Enough theory. Here is a practical framework to help you make the call for your specific site. Ask yourself these four questions.
1. Where does your traffic come from? If organic search is your main channel and you rely on Google rankings, you probably want to stay visible in AI search too. The audiences overlap more every year. If your traffic is mostly direct, social, or email-driven, AI visibility matters less and blocking hurts you minimally.
2. Is your content original and expert-level? If you publish real expertise (original research, hands-on testing, professional experience), you are more likely to get cited as a source by retrieval-based AI tools like Perplexity. That makes allowing those crawlers worthwhile. If your content is mostly curated summaries or thin content, AI crawlers probably will not cite you anyway. Blocking costs you nothing.
3. How do you make money? If your revenue is tied to direct clicks (display ads, Amazon affiliate links, product sales), AI answers that summarize your content without sending traffic hurt you. Blocking is probably the right move. If you monetize through brand authority, consulting, or lead generation, being visible everywhere, including AI search, is a net win.
4. Can your hosting handle the load? If you are on quality managed WordPress hosting that handles traffic and bot load easily, this is a non-issue. If you are on a budget shared plan that already groans under normal traffic, every AI crawler adds strain. Know your limits before you decide.
A reasonable default for most sites: do not block everything blindly. Block the pure training crawlers that give you nothing in return (GPTBot, CCBot, Amazonbot). Keep the retrieval crawlers that might send traffic (PerplexityBot, Google-Extended). Monitor your server logs and analytics, and adjust after three months. This middle-path approach protects your server while keeping you visible where it counts.
How to Block AI Scrapers (or Unblock Them)
Once you have made your decision, implementing it takes minutes. Here are your options, from simplest to most powerful.
Option 1: robots.txt Rules
This is the easiest method. Add a few lines to your robots.txt file, and compliant crawlers will obey. Most WordPress sites already have a robots.txt file. You can edit it directly or use an SEO plugin like Yoast or Rank Math to manage it.
Here is what blocking the main training crawlers looks like:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: anthropic-ai
Disallow: /
If you want to keep PerplexityBot and Google-Extended but block everything else, just add the blocking rules for the bots you do not want and leave the rest open. Check out our robots.txt guide for a complete breakdown of every AI crawler user agent and what it does.
A word of caution: robots.txt is a request, not an enforcement mechanism. A malicious crawler ignores it entirely. Most major AI companies comply, but there is no legal obligation. Treat robots.txt as your first line of defense, not your only one.
Option 2: Cloudflare WAF Rules
If your site is behind Cloudflare, you can block AI bots at the network edge before they even reach your server. This is more reliable than robots.txt because it enforces the block rather than requesting compliance. Cloudflare’s managed WAF rules now include AI bot categories, and you can also write custom rules based on user agent strings. For a full walkthrough, see our Cloudflare WAF rules guide.
Option 3: Security Plugins
WordPress security plugins like Wordfence let you block specific user agents directly. This enforces the block at the WordPress level. It is effective and does not require Cloudflare. You sacrifice the edge-level performance benefit but gain simplicity. If you are weighing plugin vs. network-level blocking, our Cloudflare vs Wordfence comparison walks through the tradeoffs.
Common Mistakes When Blocking AI Scrapers
Blocking everything, including the bots that might help you. The biggest mistake I see is site owners who add every AI user agent to their blocklist without checking what each one does. Google-Extended, for example, controls whether your site appears in Google AI Overviews. Block it and you disappear from one of the fastest-growing traffic sources. Block PerplexityBot and you lose all potential referral traffic from the AI tool that actually links back. Be selective.
Relying on robots.txt alone and assuming the job is done. robots.txt is polite but optional. If you genuinely want to enforce a block, use a WAF rule or a security plugin. The robots.txt file is your sign on the door. The WAF is your lock.
Blocking based on fear instead of data. Before you start blocking, check your server logs. Are AI crawlers actually hitting your site hard? Or are you reacting to a hypothetical problem? Use real data to decide. If your hosting is handling current traffic fine and your site is not getting hammered, aggressive blocking may be unnecessary.
Forgetting to unblock later. Your strategy may change. Maybe AI search sends real traffic next year. Maybe you change your monetization model. If you set up hard blocks through WAF rules or plugins, document them. Three months from now you should be able to re-evaluate and unblock if needed without guessing where your rules live.
Check out our Cloudflare vs Wordfence comparison — see which tool actually stops AI scrapers without slowing your site down.
Most major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) do respect robots.txt directives. They are voluntary protocols, not legal requirements, so a determined bad actor can ignore them. But the leading AI companies comply as a matter of policy. For real enforcement beyond polite requests, you need a WAF rule at the CDN level or a security plugin that blocks by user agent at the server level.
Start with the pure training crawlers: GPTBot (OpenAI), CCBot (Common Crawl, used by many AI training datasets), Amazonbot, and anthropic-ai (Anthropic’s older crawler). These crawl your site for training data and provide zero traffic or attribution in return. Keep PerplexityBot and Google-Extended unblocked unless you have a specific reason. PerplexityBot powers cited answers with backlinks, and Google-Extended controls AI Overviews visibility.
Blocking AI scrapers does not affect your regular Google Search rankings. Google’s main search crawler (Googlebot) is separate from Google-Extended, which only controls whether your content appears in AI Overviews and other Google AI products. Block GPTBot, ClaudeBot, and other training crawlers with zero impact on your search position. Only block Google-Extended if you specifically do not want your content in AI Overviews.
Check your server access logs for user agents like GPTBot, ClaudeBot, PerplexityBot, or anthropic-ai. If you use Cloudflare, the Security Analytics dashboard shows bot traffic by type. Most WordPress hosting dashboards also include bot traffic reports. Alternatively, install a security plugin like Wordfence, which logs every bot visit with the user agent and timestamp. If you see spikes from these crawlers during your publishing windows, they are actively scraping.
Yes. The simplest no-plugin method is editing your robots.txt file directly, which works for compliant crawlers. For enforced blocking without a plugin, you can add rules to your .htaccess file to deny specific user agents at the server level. If your site uses Cloudflare, you can create a custom WAF rule that blocks AI bot user agents at the CDN edge before they hit your server. No WordPress plugin installation required.
Final Thoughts
Blocking AI scrapers is not a yes or no question. It is a strategy decision that depends on where your traffic comes from, what kind of content you publish, how you make money, and what your hosting can handle.
For most WordPress site owners, the smart approach is selective: block the training crawlers that take without giving back (GPTBot, CCBot, Amazonbot), keep the retrieval crawlers that might send traffic (PerplexityBot, Google-Extended), and enforce the block with a WAF rule or security plugin rather than relying on robots.txt alone.
Revisit your decision every few months. The AI search landscape is changing fast. What looks like a smart block today might be a missed traffic opportunity next quarter. Check your analytics. Watch for AI referral traffic. Adjust accordingly.
And if the only reason you are considering blocking is server load, the real fix might be upgrading your hosting. A good managed WordPress host handles bot traffic without flinching, whether it comes from Googlebot or GPTBot. When your infrastructure is solid, you get to make strategic decisions about AI visibility without worrying about whether your site will survive the crawl.
About The Author
