Mozolite

Robots.txt tester

Fetch a site's robots.txt, find its sitemap, and test whether a path is allowed for a specific crawler.

How matching works

Crawlers pick the group whose User-agent line matches them most closely, falling back to the * group. Inside a group, the longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins. This tester follows those same rules.

Common mistakes

A leftover Disallow: / from a staging site, blocking CSS and JavaScript files that Google needs to render the page, or a Sitemap line that points to a URL returning 404.

Want the full picture?

This tool covers one thing. A Mozolite scan runs up to 201 checks on how search engines find, read and rank the page, in about 20 seconds.

Run a free scan

More free tools

Security headers checker

See which HTTP security headers a page sends, which ones are missing, and the exact line to add.

Open

Meta tags and Open Graph checker

Check the title, description, canonical and social tags of any page, and preview how it looks when shared.

Open

SSL certificate checker

Check whether a site's certificate is valid and trusted, who issued it, which TLS version it uses and when it expires.

Open
Robots.txt tester | Mozolite