OneTaskBy Tim King · Updated
Robots.txt Sensitive Path Checker
Disallow Rules That Advertise Admin Panels, Backups and Config Files to Attackers
OneTask is free, with no sign-up, and built by one person.
If this tool saved you time, a coffee helps keep new ones coming.
How the Robots.txt Sensitive Path Checker works
This robots.txt sensitive path checker looks for Disallow rules that point straight at the parts of your site you would rather keep quiet. Paste your robots.txt and it ranks each path by risk, from admin panels and backups to staging copies and internal APIs.
Why the robots.txt sensitive path checker matters
Robots.txt is a public file that anyone can read at /robots.txt. It asks well behaved crawlers to stay away from certain paths, but it does not block anyone. Attackers read it first, because a line such as Disallow: /old-admin/ or Disallow: /backup/ is a map of the places the site owner considered private.
Each Disallow path is matched against common patterns. High risk covers version control folders, configuration and environment files, backups and database dumps, database admin tools, and names such as private or secret. Medium covers admin areas, staging and test copies, logs, and folders of uploads or customer documents. Low covers API paths and file extensions that reveal the server technology.
The fix for a flagged path is to protect it properly with a login, an IP allowlist or by removing it from the server, then delete the line from robots.txt. If you just want a page out of search results, use a noindex tag instead, which does not advertise it.
FAQ
Does robots.txt stop people visiting a page?
No. It is a request that search engines and polite bots follow. Browsers, scrapers and attackers ignore it completely, so anything listed must be protected in another way.
Should I remove my admin path from robots.txt?
If the admin area is protected by a strong login, listing it adds little risk, but it also does little good. Most admin paths are never linked, so search engines rarely find them. Removing the line and adding a noindex header is the quieter option.
Why is my admin path only medium risk?
An admin area behind a strong login is a target, not a leak. Listing it tells attackers where to aim password guessing, but it does not hand them data the way an exposed backup or config file would, so those are ranked higher.
See also: all Security & Identifiers tools, Dev Utilities, Formatters & Validators, Data & Encoding, SEO, Accessibility, AI.
Related tools
Robots.txt Checker
Check a robots.txt file line by line before you deploy it.
Security Headers Checker
Check pasted response headers for missing, weak and leaky security headers.
HTTP Header Inspector
Parse pasted HTTP headers, explain each one and audit security.
Security Headers Generator
Generate HTTP security headers for nginx, Apache, Netlify and Express.
.env File Checker
Lint a .env file for syntax errors, duplicates and weak secrets.