What does Disallow: / do?

It blocks everything for that user-agent. Use Block all only when you mean it.

Will Google still index a blocked URL?

robots.txt stops crawling, not always indexing if the URL is linked elsewhere. Use noindex on the page when you need that.

Can I add more user-agent groups?

This form emits one group from the fields on screen. Add extra groups by hand after you copy, then re-test.

What is the WordPress preset?

User-agent * with Allow: /, Disallow: /wp-admin/, and Disallow: /wp-includes/.

Why is there no Test button?

This page only builds text. The preview updates as you type. Host the file at /robots.txt, then open robots.txt Tester and click Test on the live origin. Nothing here is fetched.

What does Allow all write?

User-agent: * and Allow: /. Block all writes Disallow: / and no Allow. Paths are comma-separated in Allow (comma paths) and Disallow (comma paths). Empty path slots are omitted.

Can I add Crawl-delay?

Not on this form. The preview has User-agent, Allow, Disallow, and optional Sitemap only. No comment lines. Add Crawl-delay by hand if you need it, knowing Google ignores that token.

robots.txt Generator

Loading tool…

About robots.txt Generator

Pick Allow all, Block all, or WordPress, then edit User-agent, Allow paths, and Disallow paths (comma-separated). Add an absolute Sitemap URL if you have one. The live preview updates as you type. Copy the file and host it at https://yourdomain/robots.txt.

Allow all writes User-agent: * and Allow: /. Block all writes Disallow: /. WordPress allows / and disallows /wp-admin/ and /wp-includes/. One group in this form. Crawl-delay is not on the form. The output has no comment lines. After you deploy, use the robots.txt tester against production.

When to use it

Use it when you are placing the first robots.txt on an origin, or replacing a file that grew comments and leftover groups you no longer understand. Staging and production should not share a Block all file.

Use Block all only on a staging host you intend to hide from every listed agent. Use WordPress when you want /wp-admin/ and /wp-includes/ disallowed and / allowed. Then Test the live file; this page never fetches.

How to use robots.txt Generator

1Pick a preset or start from Allow all. Allow all, Block all, or WordPress.
2Edit allow and disallow paths. Comma-separated paths. User-agent defaults to *.
3Add a sitemap URL. Absolute. Newlines are stripped from that field.
4Copy the file. Host it at https://yourdomain/robots.txt, then test it with the robots.txt tester.

How to read the results

The preview is the file you will copy: User-agent, Allow, Disallow, and optional Sitemap. Allow all is User-agent: * and Allow: /. Block all is Disallow: / with an empty Allow list. WordPress keeps Allow: / and adds Disallow: /wp-admin/ and Disallow: /wp-includes/. Copy the preview.

Comma paths become one directive per path. A blank Sitemap URL omits that line. There is no Allowed chip here. After you host the file, robots.txt Tester is where Allowed versus Disallowed appears.

Why this matters

Disallow: / for User-agent * tells every matching bot not to download any path. Linked URLs can still appear in results without a snippet, because robots.txt is not noindex. That is why Block all on production is a common outage.

The form emits one group. A GPTBot group you add by hand after Copy does not inherit the * Allow and Disallow lines. Repeat every rule you still want on that named group. Sitemap: is a hint line, not a crawl of that XML, and it does not replace a hosted sitemap file.

Limits and privacy

User-agent, Allow, Disallow, and Sitemap URL never leave this browser. There is no Test, no fetch, and no hourly cap. Copy does not publish the file. No login.

Common mistakes

Clicking Block all on the live shop and copying it to the root. The preview will show Disallow: / with no Allow.

Leaving Sitemap URL as a relative path. The field is for an absolute address. Newlines are stripped.

Expecting two User-agent boxes. One group is on the form. Extra groups are pasted by hand, then checked in robots.txt Tester against a real path and agent.

FAQs

What does Disallow: / do?: It blocks everything for that user-agent. Use Block all only when you mean it.
Will Google still index a blocked URL?: robots.txt stops crawling, not always indexing if the URL is linked elsewhere. Use noindex on the page when you need that.
Can I add more user-agent groups?: This form emits one group from the fields on screen. Add extra groups by hand after you copy, then re-test.
Does the file include comments?: No. Only User-agent, Allow, Disallow, and optional Sitemap lines.
What is the WordPress preset?: User-agent * with Allow: /, Disallow: /wp-admin/, and Disallow: /wp-includes/.
Why is there no Test button?: This page only builds text. The preview updates as you type. Host the file at /robots.txt, then open robots.txt Tester and click Test on the live origin. Nothing here is fetched.
What does Allow all write?: User-agent: * and Allow: /. Block all writes Disallow: / and no Allow. Paths are comma-separated in Allow (comma paths) and Disallow (comma paths). Empty path slots are omitted.
Can I add Crawl-delay?: Not on this form. The preview has User-agent, Allow, Disallow, and optional Sitemap only. No comment lines. Add Crawl-delay by hand if you need it, knowing Google ignores that token.