Why use this builder instead of the general robots.txt generator?

This page writes named AI groups only. Search tokens and training tokens are labeled separately. The general generator is for User-agent: * and Googlebot. Do not replace the whole file with this output.

Does generating a file change the live site?

No. Fill from URL only reads /robots.txt. The output is groups to add. You still paste and deploy the file, then re-check it with a tester.

Why is User-agent: * missing from the output?

On purpose. Named AI groups are emitted so Googlebot stays on the * rules you already have. This builder never writes a wildcard group.

Where do path rules go on a blocked bot?

They do not. A Blocked token gets Disallow: / only. Put folder rules on bots you are still admitting.

What does Allow search, block training actually flip?

Training tokens go to Block: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot. Search and Claude-User stay Allow. That is the preset only, not a live-site change.

Why are ChatGPT-User and Perplexity-User missing?

Those clients may ignore robots.txt. Listing them would imply a control you do not have. Claude-User is included because Anthropic documents that it honors the file.

What if Fill from URL finds no robots.txt?

You get No robots.txt on this site. Every crawler starts on Allow. A 404 is not treated as a site-wide Disallow: /.

Does the conflict warning mean I should delete Googlebot?

No. Conflicts are only for the ten AI tokens already named in the live file. Googlebot stays on your existing * rules.

AI Robots.txt

Loading tool…

About AI Robots.txt

AI Robots.txt writes named Allow and Block groups for ten AI tokens. Toggles start on Allow (GEO). Use Fill from URL to read the live /robots.txt, or stay on the defaults and copy the groups. Generating a file does not change the live site.

Allow search, block training flips training tokens to Block and leaves search plus Claude-User on Allow. Path rules (Disallow or Allow, placeholder /admin/) are copied into every Allow group. Blocked bots get Disallow: / only. Identical rule sets collapse into one User-agent list. A Sitemap field is omitted if that loc is already in the live file. User-agent: * is never emitted.

A default Allow run collapsed all ten tokens onto Allow: /. The search/training split plus Disallow /drafts/ emitted two groups: training tokens with Disallow: /, and OAI-SearchBot / PerplexityBot / Claude-SearchBot / Claude-User with Allow: / and Disallow: /drafts/. ChatGPT-User and Perplexity-User are omitted on purpose.

When to use it

Use it when you want search crawlers in and training crawlers out, or the reverse, without rewriting Googlebot. Use it after a rights decision, not after a heading edit. Fill from URL first if a live file already names any of these tokens, so conflicts show before you paste.

How to use AI Robots.txt

1Optional: Fill from URL. Fetches /robots.txt on that origin. A missing file shows the warning banner and leaves every toggle on Allow.
2Set Allow (GEO) or Block per bot. Or click Allow search, block training. Labels show company and role: search / citation, training, or user fetch.
3Add Path rules if you need them. Disallow or Allow plus a path. They apply to Allow groups only. Add another row for a second path.
4Copy or download the groups. Paste them into the existing robots.txt. Then confirm with the robots.txt tester or AI Crawler Access.

Why this matters for AI search

Named robots groups do not inherit each other's lines. If GPTBot has its own block, you must repeat Allow or Disallow on OAI-SearchBot if you want the same paths there. A wildcard User-agent: * rule is not copied onto these groups, which is why this builder leaves that group untouched and refuses to emit *.

That repetition is the whole point. Citation products honor OAI-SearchBot and PerplexityBot. Training products honor GPTBot and ClaudeBot. One shared * group cannot express that split. The emitted comments say Googlebot is not in this list, and that Google-Extended is a token, not a second Google crawler.

What this tool cannot do

It cannot publish the file, prove a bot obeyed it, or speak for ChatGPT-User. Fill from URL fetches /robots.txt only. Generation runs locally. No hourly cap unless you fill. Localhost is blocked. Do not paste this output over User-agent: *.

Common mistakes

Replacing the entire robots.txt with the output. The comments tell you to add groups. A full replace can drop Googlebot rules you already have.

Adding path rules and expecting them on Blocked bots. Those rows are Disallow: / only.

Ignoring a conflict chip after Fill from URL. If GPTBot is already named, adding a second group fights the first.

Treating the red warning as a proven de-index. The banner says blocking GPTBot and ClaudeBot removes content from those tools. We cannot verify that. It is a caution, not a receipt.

FAQs

Why use this builder instead of the general robots.txt generator?: This page writes named AI groups only. Search tokens and training tokens are labeled separately. The general generator is for User-agent: * and Googlebot. Do not replace the whole file with this output.
Does generating a file change the live site?: No. Fill from URL only reads /robots.txt. The output is groups to add. You still paste and deploy the file, then re-check it with a tester.
Why is User-agent: * missing from the output?: On purpose. Named AI groups are emitted so Googlebot stays on the * rules you already have. This builder never writes a wildcard group.
Where do path rules go on a blocked bot?: They do not. A Blocked token gets Disallow: / only. Put folder rules on bots you are still admitting.
What does Allow search, block training actually flip?: Training tokens go to Block: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot. Search and Claude-User stay Allow. That is the preset only, not a live-site change.
Why are ChatGPT-User and Perplexity-User missing?: Those clients may ignore robots.txt. Listing them would imply a control you do not have. Claude-User is included because Anthropic documents that it honors the file.
What if Fill from URL finds no robots.txt?: You get No robots.txt on this site. Every crawler starts on Allow. A 404 is not treated as a site-wide Disallow: /.
Does the conflict warning mean I should delete Googlebot?: No. Conflicts are only for the ten AI tokens already named in the live file. Googlebot stays on your existing * rules.