Robots.txt Validator

How the Robots.txt Checker works

Robots.txt validator: matching rules

Robots.txt is the smallest file on a site and among the most consequential. It has no error reporting: a crawler that does not understand a line simply skips it, so a broken file behaves like a working one right up until you notice traffic has gone. Reading it carefully before deploying is the only feedback loop there is.

The file is a set of groups, and order inside a group matters. A User-agent line opens a group, and every Allow and Disallow beneath it belongs to that group until the next User-agent appears. A rule placed above the first User-agent line belongs to nothing and is ignored entirely — one of the easiest mistakes to make when editing an existing file from the top.

A crawler obeys exactly one group: the most specific one that names it. If your file has a group for Googlebot and a group for *, Googlebot reads its own group and ignores the catch-all completely. That surprises people who expect the rules to combine, and it means a Disallow you added to the * group does not apply to any crawler with a named group of its own. Anything you want to apply to a specific bot has to be repeated in that bot's group.

Disallow blocks crawling, not indexing, and the difference matters. A blocked URL can still appear in search results if other pages link to it, because the crawler never fetches the page and so never sees a noindex instruction on it. If you want a page kept out of the index, it must be crawlable and carry a noindex meta tag or X-Robots-Tag header. Blocking it in robots.txt guarantees the opposite of what you wanted: the URL stays listed and its content stays unknown.

Robots.txt is not a security control. The file is public by definition, it is one of the first things any scanner fetches, and listing /admin or /internal-reports in it is an index of exactly what you would rather nobody found. Anything that must not be reached needs authentication. Use robots.txt for crawl budget and duplicate content, never for secrecy.

Path matching is prefix-based with two special characters. Every rule matches any URL starting with the given path, so Disallow: /print also blocks /printers. An asterisk matches any sequence, and a dollar sign anchors the match to the end of the URL — so Disallow: /*.pdf$ blocks PDF files specifically. Between two conflicting rules, the more specific — the longer path — wins, regardless of which appears first.

Generating a file. Switch to generate mode and the text box becomes a list of paths, one per line, that you want disallowed — leave it empty for a file that only sets the default policy and the sitemap. Paths that are missing their leading slash get one. Blocking AI training crawlers adds a group that disallows everything for GPTBot, ClaudeBot, CCBot, Google-Extended and the other named scrapers, while leaving search crawlers on the default policy. It is a courtesy request, not a wall, but the major operators publish that they honour it.

Sitemap lines stand outside the group system. They must be absolute URLs including the scheme, they can appear anywhere in the file, and they apply to every crawler regardless of the groups around them. A relative sitemap path is one of the most common errors in real files and is silently ignored.

Related tools