Robots.txt tester
Fetch a site's robots.txt, find its sitemap, and test whether a path is allowed for a specific crawler.
How matching works
Crawlers pick the group whose User-agent line matches them most closely, falling back to the * group. Inside a group, the longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins. This tester follows those same rules.
Common mistakes
A leftover Disallow: / from a staging site, blocking CSS and JavaScript files that Google needs to render the page, or a Sitemap line that points to a URL returning 404.
Want the full picture?
This tool covers one thing. A Mozolite scan runs up to 201 checks on how search engines find, read and rank the page, in about 20 seconds.