---
title: "Robots.txt tester"
description: "Free robots.txt tester. Check if a URL is blocked for Googlebot or any user agent, and validate the sitemap listed in robots.txt."
url: "https://mozolite.com/tools/robots-txt"
---

# Robots.txt tester

Fetch a site's robots.txt, find its sitemap, and test whether a path is allowed for a specific crawler.

Use the tool: https://mozolite.com/tools/robots-txt

## How matching works

Crawlers pick the group whose User-agent line matches them most closely, falling back to the * group. Inside a group, the longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins. This tester follows those same rules.

## Common mistakes

A leftover Disallow: / from a staging site, blocking CSS and JavaScript files that Google needs to render the page, or a Sitemap line that points to a URL returning 404.
