Robots.txt, Sitemaps and AI Crawlers: Controlling How Bots Read Your Site
Crawl control is three files and five decisions. Here's the correct WordPress robots.txt, a clean sitemap setup, and how to handle AI crawlers deliberately.
Every bot that visits your site — Googlebot, Bingbot, GPTBot, scrapers — starts with the same question: what am I allowed to read? You answer it with robots.txt, your sitemap, and a few meta directives. Get these three right and crawl control is done; get them wrong and you can deindex yourself with one line.
robots.txt: small file, huge blast radius
It lives at https://yoursite.com/robots.txt. WordPress serves a virtual one by default; you can override it with a physical file or a filter.
A sane baseline for most WordPress sites:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap.xml
That’s genuinely it. The classic mistakes:
Disallow: /left over from staging. The site-killer. Check after every migration.- Blocking
/wp-content/or CSS/JS. Google must render your pages; blocking assets breaks rendering and hurts rankings. - Using robots.txt to “hide” private pages. It’s a public file — you’re publishing a map of what you want hidden. Use
noindexor authentication instead. - Forgetting the
Sitemap:line. It’s the cheapest discovery hint there is.
One nuance people miss: robots.txt controls crawling, not indexing. A blocked URL can still appear in results (URL-only) if others link to it. To keep something out of the index, allow the crawl and serve a noindex meta tag.
XML sitemaps: your table of contents
WordPress has shipped native sitemaps since 5.5 at /wp-sitemap.xml (SEO plugins replace it at /sitemap_index.xml). Rules for a clean one:
- Only canonical, indexable, 200-status URLs. No redirects, no noindexed pages, no paginated archives.
- Include posts and pages; exclude tag/author/date archives you’ve noindexed (that pruning logic is part of our technical SEO checklist).
- Submit it once in Google Search Console, then just monitor the Pages report for “Discovered — currently not indexed” spikes.
AI crawlers: make a decision, don’t just default
A new generation of bots reads the web to train models and answer questions: GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training), CCBot (Common Crawl), Bytespider (ByteDance). They respect robots.txt, which means the choice is yours — and it’s a real trade-off:
- Allow them and your content can be cited in AI answers — an emerging traffic and brand channel, increasingly important as search shifts toward AI summaries.
- Block them and you protect content from training use, at the cost of invisibility in those answers.
Blocking looks like this:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Note: Google-Extended only affects Gemini training — it does not change your Google Search ranking or AI Overviews eligibility. For most businesses that live on visibility, our advice is to allow answer-engine bots and decide per-case on pure training crawlers. Whatever you choose, choose it consciously and write it down.
Meta robots and X-Robots-Tag
Page-level control belongs in the page, not robots.txt:
noindex, follow— for thin archives, search results, thank-you pages.X-Robots-TagHTTP header — same power for PDFs and feeds.
WordPress core + a well-built theme can set all of this without an SEO plugin.
Verify in 30 seconds
Our Site Analyzer checks that your robots.txt exists and doesn’t block the whole site, that a sitemap is discoverable, and reads your X-Robots-Tag — alongside 40+ other checks. Run it after every migration and redesign; the one time it catches a stray Disallow: /, it pays for itself forever (and it’s free anyway).