Robots.txt Tester

By the 1stIdeaWeb SEO team Updated 2026-11-17 2 min read

Use the Robots.txt Tester

Check how a path matches the Allow and Disallow rules in a robots.txt file you paste. The result shows the selected crawler group and the most specific matching rule.

Results will appear here after you run the tool.

Check how a path matches the Allow and Disallow rules in a robots.txt file you paste. The result shows the selected crawler group and the most specific matching rule.

How to use this tool

Copy the current robots.txt contents, choose the crawler name, and enter a path on the same host, including its query string if relevant. Run the check and review the matching rule. Test important URLs such as CSS, JavaScript, and canonical page paths.

How to understand the result

This client-side checker applies common robots.txt group, wildcard, end-anchor, and longest-match behavior. A longer matching rule generally takes precedence; an Allow rule wins an equal-length tie. The page reports the matching directive so you can explain the outcome.

Scope and limitations

Robots.txt is a crawl directive, not a privacy or access-control system. This simplified checker is not an official Google tester and may not cover every parser edge case or every crawler’s interpretation. A robots.txt Disallow does not remove an already indexed URL; use the correct page-level indexing controls for that.

Your input stays in your browser

Processing happens in this page in your browser. The tool does not send the text you enter to 1stIdeaWeb or save it on our server. Avoid entering passwords, private customer information, or other sensitive data into any public website.