robots.txt Tester
A tool that checks whether URLs may be crawled based on robots.txt contents and a user agent, following RFC 9309. It shows the group, rule, and line number that applied, so you can verify conflicting Allow/Disallow rules, wildcards (*), and end anchors ($). No site is accessed — everything runs in your browser.
- Matching follows RFC 9309 (Robots Exclusion Protocol). The group is chosen by an exact, case-insensitive match on the product token, falling back to the * group if none matches. Groups for the same user agent are combined.
- Rules are matched from the start of the URL path (including the query), and the longest match wins. When an Allow and a Disallow of the same length conflict, Allow wins. * matches any characters and a trailing $ marks the end of the URL.
- Only crawling is checked. An allowed URL isn't guaranteed to appear in search results, and a disallowed URL may still be listed (without content) if other pages link to it. Use noindex to keep a page out of results.
Result
Enter robots.txt contents and URLs to see whether each one may be crawled.
How to use
- Paste the robots.txt contents or load the file.
- Enter a user agent (e.g. Googlebot) and the URLs or paths to check, one per line.
- Each URL shows whether crawling is allowed, plus the rule and line number that applied.
- Directives not used for matching, such as Crawl-delay, and lines that can't be parsed are listed below.
About this tool
A mistake in robots.txt can stop public pages from being crawled or expose admin pages to crawlers. This tool lets you check that URLs are allowed or disallowed as intended before publishing.
Matching follows RFC 9309: the group matching the user agent (or the * group if none) is chosen, and the rule with the longest match on the URL path wins. When an Allow and a Disallow of the same length conflict, Allow wins.
Crawling and appearing in search results are separate. A disallowed page can still be listed by URL if other sites link to it; use noindex to keep it out. No site is accessed and robots.txt is never fetched automatically.
Enter three lines: User-agent: *, Disallow: /admin/, and Allow: /admin/help. Set the user agent to Googlebot. /admin/settings is disallowed, while /admin/help is allowed by the Allow rule on line 3.
The /admin/help rule is a prefix match, so it also matches /admin/helpful. Add $ at the end to target an exact ending, and test URLs with query strings too. Entering a URL does not access the site; matching uses the pasted rules and path text.
Frequently asked questions
What should I enter as the user agent?
Enter the crawler's product token, such as Googlebot or Bingbot — not a full browser user-agent string. Case doesn't matter.
What if no rule matches?
It's treated as allowed. The same applies when no group matches the user agent.
Is Crawl-delay used?
No. It's outside RFC 9309, so it's listed with its line number as a directive not used for matching.