WordPress robots.txt Generator with AI Crawler Toggles
Including the AI crawlers nobody else lets you toggle
Runs in your browser. Nothing is uploaded.
The default WordPress robots.txt
If there is no real robots.txt file at your web root, WordPress prints this one on the fly (for a site installed at the domain root, with pretty permalinks and search engines allowed):
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlReplace example.com with your domain. The Sitemap line has been added by core since WordPress 5.5, and only while the site is public. Without pretty permalinks it points at /?sitemap=index instead.
About the WordPress robots.txt Generator with AI Crawler Toggles
Generate a WordPress robots.txt with the standard rules — and per-bot toggles for GPTBot, ClaudeBot, PerplexityBot, CCBot, and Google-Extended. Decide what trains on your writing, one checkbox at a time.
When you need it
Most sites do fine with the default file shown at the top of this page. Generate your own when you want rules core does not add: keeping AI crawlers like GPTBot out, skipping internal search pages, or pointing at a sitemap from an SEO plugin instead of wp-sitemap.xml.
How to use it
- Tick the rules you want and enter the sitemap URL for the host you actually use (www or not).
- Save the output as a plain text file named robots.txt.
- Upload it to your web root, the same folder as wp-config.php on most installs. A real file takes over from the virtual one.
- Open yoursite.com/robots.txt in a browser to confirm the new rules are live.
Common mistakes
- Using robots.txt to hide a page from Google. Google says it is not a way to keep a page out of its index; use noindex or a password instead.
- Blocking /wp-content/ or /wp-includes/. Google asks you not to block CSS and scripts it needs to understand the page.
- Expecting "Discourage search engines" to write Disallow: / into the file. Since WordPress 5.3 that setting adds a noindex robots meta tag instead.
The WordPress robots.txt best practice guide explains each default line, and how to block AI crawlers covers what the bot toggles do and do not stop.
Related
Other tools you might need
FAQ
Questions
Does blocking Google-Extended hurt my search ranking?
No. Google-Extended controls whether your content is used to train Gemini. It has no effect on Googlebot, on indexing, or on your position in Google Search. This is the single most common misunderstanding about the AI crawler directives.
Will robots.txt actually stop AI companies scraping my site?
It stops the ones that respect it, which is most of the named ones. It is a request, not a fence — a crawler that ignores robots.txt is entirely unaffected. Anyone selling robots.txt as protection against scraping is overstating it.
Where does the robots.txt file go?
In your site's web root, alongside wp-config.php, so it resolves at example.com/robots.txt. WordPress serves a virtual robots.txt when no real file exists; uploading a real one takes over from it.