Is this Search Console's tester?

No. It uses a robots parser on the file we fetched. Google's own tester can still differ on edge cases.

Why did /api become /api/?

Extensionless paths get a trailing slash so a directory Disallow: /api/ matches. Files with a dot are left as you typed them.

What if robots.txt is 404?

We report 404 and treat the path as allowed. That is the usual default when no file exists.

Does Google honor Crawl-delay?

Google ignores it. Other crawlers may use it. We still display the value when it is in the file.

How is the user-agent group picked?

Longest listed token that appears inside the agent string, except *. Googlebot-Image beats Googlebot when both are listed.

Why was Wikipedia Special:Search Disallowed for Googlebot?

The file returned 200. The winning rule was Disallow: /wiki/Special: on line 156, and the matching group was *. The chip is Disallowed (critical). The same host on path / is Allowed. Presets do not force a named group if * is the one that lists the rule.

What user-agent presets exist?

*, Googlebot, Googlebot-Image, Bingbot, GPTBot, and CCBot. Choose Custom to type another token. The parser then picks the longest listed User-agent that is a substring of what you typed, skipping * unless nothing else matches.

Does Test crawl the site?

No. It fetches robots.txt for that origin and evaluates one path. Sitemap: lines are listed, not opened. Indexability Checker is the page that also reads meta robots and X-Robots-Tag.

robots.txt Tester

Loading tool…

About robots.txt Tester

Enter a site origin or a robots.txt URL, a path, and a user-agent, then click Test. Presets: *, Googlebot, Googlebot-Image, Bingbot, GPTBot, and CCBot, or type a custom agent. You get Allowed or Disallowed, the matching group, HTTP status, optional Crawl-delay, the winning rule with line number, the full file with that line highlighted, Sitemap: URLs, and syntax issues.

This is not Google Search Console's official tester. It uses a real robots parser. A 404 robots.txt is treated as allow all. An HTML body instead of a robots file is a warning and we evaluate empty rules.

When to use it

Use it after you edit robots.txt, when a path that should stay out of the index is still being fetched, or before you submit a sitemap that lists a URL you just Disallowed.

Also use it when GPTBot and Googlebot need different answers and you must see which User-agent group actually won. It is not Search Console's official tester and it does not crawl the site.

How to use robots.txt Tester

1Enter the site origin or robots URL. example.com is enough; we request /robots.txt.
2Enter a path and user-agent. Directory names like /api are tested as /api/. Pick a preset or type a custom agent.
3Click Test. Allowed versus Disallowed plus the winning rule.
4Fix robots.txt on the origin. Then Test again.

How a rule is chosen

The matching group is the longest listed User-agent token that is a substring of the agent you tested (case-insensitive), skipping *. Googlebot-Image wins over Googlebot when both groups exist. Empty path can inherit the site pathname. Extensionless paths such as /api are tested as /api/ so a Disallow: /api/ directory rule applies. Paths with a file extension are left as typed.

Crawl-delay is shown when present. Google ignores Crawl-delay; other bots may honor it. Allowed means the parser did not return a disallow for that agent and URL.

How to read the results

The chip is Allowed or Disallowed. You also get HTTP status of the robots file, the matching group, optional Crawl-delay, the winning rule with its line number, Sitemap: URLs, syntax issues, and the full file with that line highlighted. Progress on Test is Reading robots.txt.

example.com on / with * is Allowed plus No robots.txt (404). Wikipedia /wiki/Special:Search with Googlebot is Disallowed: line 156, Disallow: /wiki/Special:, group *. The homepage path / on the same host is Allowed.

Why this matters

A bot that matches a Disallow never downloads that document, so later on-page tags on that URL are invisible to it. That is useful for /wiki/Special: search UI and fatal when you Disallow a section you still listed in a sitemap.

The winning group is not always the preset you picked. Wikipedia's Googlebot test still matched the * group because that is where /wiki/Special: is listed. Named groups do not inherit * rules, which is why a Googlebot-only Allow can still lose if a longer token matches first.

Limits and privacy

Fields are Site or robots.txt URL, a path, and a user-agent. Loopback and reserved IPs are rejected. About 20 Tests per minute on your network. Only robots.txt is fetched. No login.

Common mistakes

Testing /api without the slash and thinking a Disallow: /api/ missed. Extensionless paths are tried as /api/.

Reading Allowed on a 404 file as a custom Allow rule. No robots.txt means allow all.

Assuming Google honors Crawl-delay because the field is shown. Google ignores it. Other agents may not.

FAQs

Is this Search Console's tester?: No. It uses a robots parser on the file we fetched. Google's own tester can still differ on edge cases.
Why did /api become /api/?: Extensionless paths get a trailing slash so a directory Disallow: /api/ matches. Files with a dot are left as you typed them.
What if robots.txt is 404?: We report 404 and treat the path as allowed. That is the usual default when no file exists.
Does Google honor Crawl-delay?: Google ignores it. Other crawlers may use it. We still display the value when it is in the file.
How is the user-agent group picked?: Longest listed token that appears inside the agent string, except *. Googlebot-Image beats Googlebot when both are listed.
Why was Wikipedia Special:Search Disallowed for Googlebot?: The file returned 200. The winning rule was Disallow: /wiki/Special: on line 156, and the matching group was *. The chip is Disallowed (critical). The same host on path / is Allowed. Presets do not force a named group if * is the one that lists the rule.
What user-agent presets exist?: *, Googlebot, Googlebot-Image, Bingbot, GPTBot, and CCBot. Choose Custom to type another token. The parser then picks the longest listed User-agent that is a substring of what you typed, skipping * unless nothing else matches.
Does Test crawl the site?: No. It fetches robots.txt for that origin and evaluates one path. Sitemap: lines are listed, not opened. Indexability Checker is the page that also reads meta robots and X-Robots-Tag.