How to Block AI Crawlers in WordPress (Honestly)

To stop GPTBot from crawling a WordPress site, add these lines to your robots.txt:

User-agent: GPTBot
Disallow: /

That is the entire fix for well-behaved crawlers. But be honest with yourself about what you just did: robots.txt is a request, not a fence. OpenAI’s GPTBot fetches the file and obeys it. A scraper that does not respect the standard fetches the same file and ignores every line in it. Keep that distinction in front of you before you spend an afternoon tuning this.

If you want to cover the major AI crawlers in one pass, the common user-agent tokens are:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

GPTBot is OpenAI’s training crawler, CCBot is Common Crawl (which many models train on downstream), ClaudeBot is Anthropic, Google-Extended is Google’s AI-training token, and Bytespider is ByteDance. Note two of these already break the polite-request model: Bytespider and PerplexityBot have both been publicly documented crawling sites that disallowed them. Listing them still helps against the honest bots; it does nothing against the ones already lying about who they are.

How WordPress actually serves robots.txt

This is where most people’s edits silently do nothing. If there is no physical robots.txt file in your web root, WordPress generates one on the fly. The request is handled by do_robots() in wp-includes/functions.php, and the output runs through the robots_txt filter. The “Discourage search engines from indexing this site” checkbox under Settings → Reading flips the blog_public option, which is what makes that virtual file emit Disallow: /.

So you can append AI-crawler rules programmatically:

add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "User-agent: GPTBot\nDisallow: /\n";
    return $output;
}, 10, 2 );

Here is the gotcha that eats hours: the moment a static robots.txt file exists in your web root, Apache or nginx serves that file directly and WordPress never runs. The robots_txt filter, the Yoast/Rank Math robots editor, the Reading setting — all bypassed. If you edited robots.txt in a plugin and the live file never changed, this is almost always why. Check https://yoursite.com/robots.txt in a private window and compare it to what you think you set. If there is a real file on disk, edit that file; the filter is irrelevant.

Fixing it, in order

First, write clean rules. Get the syntax and the crawler list right so you are not blocking Googlebot by accident. Our robots.txt generator builds the AI-crawler block for you and shows the exact file to paste, which sidesteps the virtual-vs-physical confusion because you end up with one authoritative file.

Second, confirm it is live. Fetch the URL directly. Do not trust the plugin’s preview pane.

Third, if you need enforcement rather than a polite ask, move up the stack. robots.txt lives at the application layer and depends on the bot’s goodwill. To actually refuse a connection, block by user agent at the server. In .htaccess:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|CCBot|ClaudeBot|Bytespider) [NC]
RewriteRule .* - [F,L]
</IfModule>

That returns 403 to anything sending those user agents. Our .htaccess generator can assemble this rule. Understand its limit, though: user agents are trivially spoofed, so this only stops bots honest enough to identify themselves — which are the same bots that already obey robots.txt. The genuinely robust option is a CDN or WAF that blocks by verified crawler identity or IP range. Cloudflare’s one-click “Block AI Scrapers and Crawlers” managed rule is the least-effort version of this and works at the edge, before the request reaches WordPress at all.

What not to do

Do not block Google-Extended thinking it keeps you out of AI Overviews. This is the misconception worth correcting. Google-Extended is a product token, not a crawler. It controls only whether your content is used to train and ground Gemini models. Per Google’s own documentation, it has no effect on how your pages are crawled, indexed, or ranked in Google Search. And AI Overviews are built from the regular Search index that Googlebot crawls — not from Google-Extended. Blocking Google-Extended will not remove you from AI Overviews, and it will not cost you a single position in Search. The only way to stay out of AI Overviews is to block Googlebot or noindex the page, which also deletes you from Search entirely. That is almost never a trade you want.

Do not treat robots.txt as a security control. It is public and advisory. Listing /wp-admin/ or a private path in it just publishes a map of where to look. Protect real endpoints with authentication, not a disallow line.

Do not rely on noai / noimageai meta tags. They were proposed, not standardized, and only a handful of platforms honor them. They are not a general defense.

Do not assume a “block AI” plugin did more than write robots.txt. Most of them write the same lines you could write by hand and inherit the same limitation. Read what the plugin actually changed before trusting it.

Still stuck?

If your edits are not showing up, the answer is almost always a stray physical robots.txt in the web root overriding WordPress’s virtual one — fetch the URL directly and check. If compliant bots are gone but scrapers keep hitting you, you have hit the ceiling of what robots.txt can do, and the next move is a WAF or CDN rule that blocks by verified identity rather than by asking nicely.