Mozolite

AI crawler access checker

See whether ChatGPT, Claude, Perplexity, Gemini and other AI crawlers are allowed to read a page, and whether the site has an llms.txt file.

Why this matters now

A growing share of people ask AI assistants instead of searching. Those assistants rely on their own crawlers to read and cite pages. If robots.txt blocks them, your content cannot be quoted or linked in their answers.

Training and search are separate

Several companies use different user agents for model training and for live answers. For example OAI-SearchBot and ChatGPT-User fetch pages for answers, while GPTBot is used for training. You can allow one and block the other.

About llms.txt

llms.txt is a proposed plain-text file at the root of a site that gives language models a short summary and a list of key pages. It is not a standard yet and no major assistant has committed to reading it, so treat it as optional.

Want the full picture?

This tool covers one thing. A Mozolite scan runs up to 201 checks on how search engines find, read and rank the page, in about 20 seconds.

Run a free scan

More free tools

Security headers checker

See which HTTP security headers a page sends, which ones are missing, and the exact line to add.

Open

Meta tags and Open Graph checker

Check the title, description, canonical and social tags of any page, and preview how it looks when shared.

Open

Robots.txt tester

Fetch a site's robots.txt, find its sitemap, and test whether a path is allowed for a specific crawler.

Open
AI crawler access checker | Mozolite