Why use Crawler Access after Citation Readiness?

Citation Readiness grades one URL. This tool maps which named bots robots.txt admits to which discovered paths. Use it after you edit robots.txt, or when a section of the site should be open and is not.

Can I check more than 500 pages?

Yes. Pages to check includes 2,000. Discovery may return more URLs than you grade. Found stays at the full list. Raise the cap to evaluate more of it.

Why does it ask me to paste URLs?

robots.txt was missing or the homepage HTML had no followable links (js-shell). Paste paths on that origin, one per line. A missing file is treated as default allow for named tokens.

Does a blocked bot mean ChatGPT deleted my site?

No. The row is a robots.txt decision for that token and path today. It does not erase training data. ChatGPT-User may fetch anyway.

What does Partial mean in the counts?

At least one of the ten major tokens is blocked on that URL, and at least one is allowed. Expand the row for the winning rule and line. The other eight tokens are listed there too.

Does Google-Extended in the access matrix change Google ranking?

No. Google-Extended is Gemini and AI Overviews training. A Disallow on that token is not a Googlebot rule and does not move blue-link ranking.

Why does a longer Allow beat a shorter Disallow?

RFC 9309 longest-match. If both rules hit the path, the longer string wins. The row names both when that tie-break happens.

Does Check AI access download every article?

No. After /robots.txt and the URL list are in, each row is a path match. Page bodies are not fetched again.

AI Crawler Access

Loading tool…

About AI Crawler Access

AI Crawler Access shows which named bots your robots.txt admits to which discovered URLs. Paste any page on the site, pick Pages to check (25, 50, 100, 250, 500, or 2,000; default 100), click Check AI access, and read the per-path matrix.

The server fetches /robots.txt once, then builds a URL list from sitemap.xml (including a sitemap index and .xml.gz) or, if that fails, from homepage links. Checking is local after that: each path is matched against all 18 tokens. The header links View robots.txt and can Show file. Counts are Found, Readable, Partial, and Blocked. Filters are All, Blocked, and Readable. Blocked rows sort first. Download writes CSV.

Readable means the ten major tokens (GPTBot through CCBot) are allowed. Partial means some of those ten are blocked. The other eight (Bytespider through cohere-ai) still appear when you expand a row. This is not a live crawl of each page body.

When to use it

Use it after you edit robots.txt, and before you argue about why a model never saw a section of the site. Use it when a competitor is cited from a path you thought was public. Run it once per robots.txt change, not after every copy tweak.

How to use AI Crawler Access

1Paste any URL on the site. The origin is derived from that address. Localhost is rejected.
2Set Pages to check. 25, 50, 100 (default), 250, 500, or 2,000. Matching is instant after robots.txt is in.
3Click Check AI access. Progress reads robots.txt, then the sitemap or homepage. A js-shell result asks you to paste URLs, one per line.
4Read the counts, then expand rows. Blocked sorts to the top. Use All / Blocked / Readable and the path search. Download CSV if you need the full matrix.

How to read the results

example.com returned robots.txt HTTP 404 (missing), discovery code js-shell, and a paste box: no sitemap and no followable homepage links. Wikipedia returned HTTP 200, a found file, and 167 URLs from a homepage crawl. If Found is larger than the table, the line says Raise the limit to check more.

Expand a row for the winning rule and line number per bot. When Allow and Disallow both match, the longer path wins (RFC 9309). A named group for GPTBot is not inherited by OAI-SearchBot. User-agent: * is used only when that bot has no group of its own. A site-wide note appears if one token is blocked on every checked URL. Content-folder callouts name paths that look like articles and are closed.

Why this matters for AI search

robots.txt has no inheritance. A Disallow written under User-agent: * does not automatically apply to GPTBot if a GPTBot group exists, and a GPTBot group does not speak for OAI-SearchBot. Search crawlers and training crawlers are different tokens. You can admit OAI-SearchBot and still block GPTBot. The table is how you see that split per path, instead of guessing from the wildcard.

What this tool cannot do

It cannot prove a bot honored the file. It cannot fetch page bodies or run JavaScript while discovering links. A JavaScript homepage with no sitemap ends in a paste box. Five discovery runs per hour. Localhost is blocked. The matrix after fetch is computed in your browser from the file we already have.

Common mistakes

Reading a missing robots.txt as a block. HTTP 404 means no file, so named bots follow the usual default allow.

Stopping at the first 10 rows. The preview is 10. Raise Pages to check, then use the path search.

Treating ChatGPT-User as GPTBot. ChatGPT-User is a live user fetch and may ignore the file you just inspected.

Assuming Partial means the HTML is thin. Partial is a robots split among the ten major tokens, not a word count.

FAQs

Why use Crawler Access after Citation Readiness?: Citation Readiness grades one URL. This tool maps which named bots robots.txt admits to which discovered paths. Use it after you edit robots.txt, or when a section of the site should be open and is not.
Can I check more than 500 pages?: Yes. Pages to check includes 2,000. Discovery may return more URLs than you grade. Found stays at the full list. Raise the cap to evaluate more of it.
Why does it ask me to paste URLs?: robots.txt was missing or the homepage HTML had no followable links (js-shell). Paste paths on that origin, one per line. A missing file is treated as default allow for named tokens.
Does a blocked bot mean ChatGPT deleted my site?: No. The row is a robots.txt decision for that token and path today. It does not erase training data. ChatGPT-User may fetch anyway.
What does Partial mean in the counts?: At least one of the ten major tokens is blocked on that URL, and at least one is allowed. Expand the row for the winning rule and line. The other eight tokens are listed there too.
Does Google-Extended in the access matrix change Google ranking?: No. Google-Extended is Gemini and AI Overviews training. A Disallow on that token is not a Googlebot rule and does not move blue-link ranking.
Why does a longer Allow beat a shorter Disallow?: RFC 9309 longest-match. If both rules hit the path, the longer string wins. The row names both when that tie-break happens.
Does Check AI access download every article?: No. After /robots.txt and the URL list are in, each row is a path match. Page bodies are not fetched again.