Robots.txt & Sitemap Inspector
Validate a site’s robots.txt, test any path against real crawlers, and check the sitemaps it declares.
What is a robots.txt tester?
A robots.txt tester — also called a robots.txt validator or checker — reads the file at the root of a host and tells you what it actually permits — not what you meant it to say. /robots.txt is the oldest convention on the web for telling crawlers where they may go, standardised as RFC 9309 in 2022, and its matching rules surprise people: patterns are prefixes rather than path segments, a named user-agent group replaces the wildcard group rather than adding to it, and the longest matching rule wins with ties going to Allow.
Because /robots.txt governs a whole host, this is a domain-level check — the path you enter does not change the result, though you can test any path against the parsed rules once the file is loaded. Note that www.example.com and example.com are different hosts with different files.
Crawling is not indexing
The most expensive mistake made with this file is treating Disallow as a way to keep a page out of Google. It is not. robots.txt controls crawling; noindex controls indexing, and they are separate systems.
A disallowed URL can still be indexed — if anything links to it, a search engine may list the URL with no title and no description, because it was never permitted to fetch the page and see what is there. And the failure compounds: if that page carries a noindex meta tag, blocking the crawler means the tag is never read. The disallow you added to remove the page is the reason it stays.
The rule worth remembering: to keep a page out of an index, allow crawling and serve noindex. Use Disallow to save crawl budget on pages you do not care about, not to hide pages you want gone.
How to Use the Robots.txt Inspector
- Enter a domain or any URL on it and click Inspect robots.txt.
- Read Overview for the file’s status, how many rules it carries, and which AI crawlers it currently blocks.
- Use the Match Tester to check any path against a specific crawler. Results appear as you type, and you can check as many paths and crawlers as you like.
- Open Sitemaps to see every sitemap discovered, its format, entry count and any parse error.
- Check Conflicts for URLs a sitemap advertises that robots.txt blocks.
How the Inspector Works
We request /robots.txt over HTTPS, following redirects, and treat only a 200 carrying a text file as a real robots.txt — a 404 body and an HTML bot-wall page served with a 200 are both reported as what they are rather than parsed as rules. The file is parsed into user-agent groups following the RFC 9309 grammar, including the widely deployed Crawl-delay, Host and Clean-param extensions.
Sitemaps are discovered from Sitemap: lines, then from any sitemapindex they reference, and only if none are declared do we probe the conventional locations. Every common format is read: urlset, sitemapindex, RSS, Atom, plain text and gzipped.
One check reads a sample of a site’s sitemaps rather than all of them, and the result tells you when it did — so the counts you see always describe what was actually read.
Key Features
- RFC 9309 matching — Longest-token group selection, longest-rule precedence,
Allowwins ties, and duplicate groups for one agent merged as the RFC requires. - Live path tester — Check any path against eight built-in crawlers plus any you add, with results as you type.
- AI crawler matrix — Whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and others are allowed, with the rule that decided it.
- Full sitemap walk — Nested indexes, RSS and Atom feeds, plain-text and gzipped sitemaps.
- Conflict detection — URLs a sitemap lists that robots.txt disallows.
- Raw JSON — The complete response, for scripting and CI checks.
Common Use Cases
- SEO specialists — Confirm a new section is crawlable and correctly listed in a sitemap before launch.
- Publishers — Audit which AI crawlers your file admits, and verify a change took effect.
- DevOps engineers — Catch a staging
Disallow: /that shipped to production. - Content teams — Find sitemap entries that robots.txt quietly blocks.
Frequently Asked Questions
What does the robots.txt tester check?
It fetches /robots.txt, parses it against the RFC 9309 grammar into user-agent groups and rules, and reports the Sitemap, Crawl-delay, Host and Clean-param directives it finds. It then walks the sitemaps robots.txt declares — following sitemapindex references — samples their entries, and flags any sampled URL that a sitemap advertises while robots.txt disallows it. A path tester lets you check any URL against Googlebot, Bingbot, GPTBot, ClaudeBot and others.
Does blocking a page in robots.txt stop it being indexed?
No, and this is the single most common misunderstanding about the file. robots.txt controls crawling, not indexing. A URL you disallow can still appear in search results if other pages link to it — Google simply lists it with no description, because it was never allowed to fetch the page. Worse, if the page carries a noindex meta tag, a crawler blocked by robots.txt can never read that tag, so the disallow actively prevents the removal you wanted. To keep a page out of an index, allow crawling and use noindex.
How do I block ChatGPT, Claude and other AI crawlers?
Add a group per agent — GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google’s AI training, PerplexityBot, CCBot for Common Crawl — each with Disallow: /. The AI bots panel shows which of these your file currently allows. Note that these are voluntary: robots.txt is a request, honoured by well-behaved crawlers and ignored by others, so it is not an access control.
Why does a named group override the wildcard group?
Because RFC 9309 says exactly one group applies to a crawler: the one whose user-agent token is the longest match for its name. Groups are not merged across specificity. So if your file has Disallow: / under User-agent: * and Allow: / under User-agent: Googlebot, Googlebot may crawl everything — the wildcard rule does not apply to it at all. This trips people up constantly, which is why the tester reports which group produced each verdict.
What counts as a sitemap and robots.txt conflict?
A URL that a sitemap lists and robots.txt disallows. A sitemap is a request to crawl; a disallow is a request not to. Together they are a contradiction that wastes crawl budget and, for a page you actually want indexed, silently prevents it. The tester samples entries from each sitemap it reads and evaluates them against the wildcard group.
Is the robots.txt tester free?
Yes. Every HTTPSense tool is free, needs no signup and no API key, and returns the full result including the raw JSON.
Does it read every URL in my sitemap?
No. Each sitemap contributes a sample of its first entries, and on a site with many sitemaps only some are read. The result says so when that happens, so the counts and conflicts you see always describe what was actually checked. The sample is taken in document order rather than at random, so running the same check twice gives the same answer.
Does Crawl-delay work? Does Google support it?
Google does not support Crawl-delay and ignores it entirely. It is not part of RFC 9309 — it was a widely copied extension that never made it into the standard. Bing and Yandex do honour it, and most other crawlers are undocumented either way. So a Crawl-delay is doing something, just not what most people writing one assume: it does not slow Googlebot. To reduce Googlebot’s crawl rate, use the crawl rate setting in Search Console, or serve 503 responses, which it backs off from. The Inspector labels the directive as non-standard wherever it appears, and raises a note when a Crawl-delay sits in the group Googlebot actually matches.
Why does a rule for /admin also block /administrator?
Because robots.txt patterns are prefix matches, not path-segment matches. Disallow: /admin blocks every path beginning with those characters, including /administrator and /admin-tools. Use Disallow: /admin$ to match only that exact path, or Disallow: /admin/ to match only things beneath it.