# Robots.txt Generator

> Build a robots.txt file with Allow and Disallow rules per crawler, sitemap lines and presets for blocking AI training bots. Copy or download it.

- URL: https://generate.now/robots-txt
- Category: Config & DevOps
- Price: free, no account required
- AI-powered: no (runs in the browser)
- Last updated: 2026-11-11

## What is a robots.txt file?

robots.txt is a plain text file at the root of a website that tells crawlers which URLs they may request. It is defined by RFC 9309 and made of groups: one or more User-agent lines followed by Allow and Disallow rules. It is a request to crawlers, not access control, and it does not remove pages from search results.

## How the Robots.txt Generator works

The tool builds a robots.txt from rule groups: pick the crawlers a group applies to, add Allow and Disallow paths, and list your sitemaps. Presets cover allowing everything, blocking everything, blocking the crawlers whose operators say they collect data for AI training, and the rules WordPress serves by default. Paths, user-agent tokens and sitemap URLs are validated as you type. A plain-English summary explains what the file does, and a checker tells you whether a given URL is allowed for a given crawler using the longest-match rule from RFC 9309. The file is built in your browser and is not sent to our server.

## How crawlers decide which rule applies

A crawler first looks for a group that names its own token, compared without regard to case. If one exists it follows only that group and ignores the * group. If none does, it follows the * group. If there is no * group either, nothing is restricted.

Inside the group, every Allow and Disallow path is compared with the start of the URL path, and paths are case-sensitive. The rule with the longest matching path wins. If an Allow and a Disallow match with the same length, Allow wins. * matches any run of characters and $ marks the end of the URL. The examples are from Google's documentation and give the same result in this tool's checker.

| Rules | URL path | Result | Why |
| --- | --- | --- | --- |
| Allow: /p and Disallow: / | /page | Allowed | /p is the longer match |
| Allow: /folder and Disallow: /folder | /folder/page | Allowed | Same length, Allow wins |
| Allow: /page and Disallow: /*.htm | /page.htm | Disallowed | /*.htm is the longer match |
| Allow: /$ and Disallow: / | /page.htm | Disallowed | /$ matches only the home page |
| Disallow: /fish | /Fish.asp | Allowed | Paths are case-sensitive |

*Which rule wins*

## AI crawler user agents and what each one is for

Each row is taken from the operator's own documentation, read on 11 November 2026. The "Block AI training" preset disallows the six tokens marked Yes: the ones whose operators say the crawl may be used to train models. It leaves AI search crawlers and user-requested fetches alone, so a site can still be found and cited in AI answers.

Google-Extended and Applebot-Extended are not separate crawlers. They are tokens that control how pages already fetched by Googlebot and Applebot may be used, and both companies say that disallowing them does not affect search results. OpenAI and Perplexity say robots.txt may not apply to fetches a user asks for.

| Token | Operator | Used for | In preset |
| --- | --- | --- | --- |
| GPTBot | OpenAI | Content that may train models | Yes |
| ClaudeBot | Anthropic | Content that may train models | Yes |
| Google-Extended | Google | Use of crawled pages for Gemini training | Yes |
| Applebot-Extended | Apple | Use of crawled pages for Apple's models | Yes |
| Meta-ExternalAgent | Meta | Training AI models, indexing | Yes |
| Amazonbot | Amazon | Amazon products; may train models | Yes |
| OAI-SearchBot | OpenAI | ChatGPT search results | No |
| ChatGPT-User | OpenAI | Pages a user asks ChatGPT to open | No |
| Claude-SearchBot | Anthropic | Claude search results | No |
| Claude-User | Anthropic | Pages a user asks Claude to open | No |
| PerplexityBot | Perplexity | Perplexity search results | No |

*AI user-agent tokens*

## When to use the Robots.txt Generator

- Write the first robots.txt for a new site, with a Sitemap line
- Keep crawlers out of admin, cart and internal search URLs
- Opt out of AI training crawlers while staying visible in search
- Check whether a URL is allowed for a crawler before deploying a rule change

## Robots.txt Generator examples

### Preset: Allow all

```
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml
```

An empty Disallow matches nothing, so every crawler may fetch every URL. Replace the sitemap URL with your own.

### Preset: Block AI training

```
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
Disallow: /

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml
```

One group disallows the whole site for six tokens. Every other crawler falls through to the * group, which allows everything.

### Preset: WordPress

```
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml
```

The rules WordPress core generates. admin-ajax.php stays crawlable because its Allow path is longer than the Disallow path.

### Check /admin/public/report.html for Googlebot against Disallow: /admin/ and Allow: /admin/public/

```
Allowed. No group names Googlebot, so the * group applies. Allow: /admin/public/ is the longest match, ahead of Disallow: /admin/.
```

The checker names the group that applied and the rule that won.

## Frequently asked questions

### Does robots.txt remove a page from Google?

No. Google's documentation says robots.txt "is not a mechanism for keeping a web page out of Google". A disallowed URL can still be indexed if other pages link to it, and then appears without a description. To keep a page out of search results, let it be crawled and add a noindex robots meta tag or header, or put it behind a password.

### Where does the robots.txt file go?

It goes at the top level of the host, named exactly /robots.txt and served as UTF-8 plain text. It applies only to the protocol, host and port it is served from, so https://example.com/robots.txt does not cover shop.example.com or the http version. A file in a subdirectory is not read.

### Which rule wins when Allow and Disallow both match?

The one with the longest path. RFC 9309 says the most specific match, the one with the most octets, must be used, and that Allow should win when the two are equal. Order in the file does not matter. With Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php, admin-ajax.php is allowed and the rest of /wp-admin/ is not.

### Do crawlers with their own group also follow the * rules?

No. A crawler that finds a group naming it uses only that group. If Googlebot has its own group and you also want it kept out of /admin/, repeat Disallow: /admin/ there. The summary under the output points this out whenever your file has both named groups and a * group.

### Does Google obey Crawl-delay?

No. Crawl-delay is not part of RFC 9309, and Google's documentation lists it as a field it does not support. Bing has said it honours values from 1 to 30, Anthropic says ClaudeBot supports it, and Apple and Amazon say their crawlers do not follow it. The tool writes the line only if you fill the field in.

### How do I block AI crawlers with robots.txt?

Add a group that names their tokens and disallows /. The "Block AI training" preset does this for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and Amazonbot, the tokens whose operators say the crawl may be used for model training. robots.txt is a request: it works for crawlers that choose to honour it and does nothing against one that ignores it.

### What happens if robots.txt is missing or returns an error?

Under RFC 9309, a 4xx response such as 404 means the file is unavailable and the crawler may access everything. A 5xx response means it is unreachable and the crawler must assume the whole site is disallowed. Crawlers should not use a cached copy for more than 24 hours, and must be able to parse at least 500 kibibytes; Google ignores anything past that size.

### Are my rules sent to your server?

No. The file, the summary and the URL check are all computed in the page. The page loads standard page analytics, which record the visit and that a preset, Copy or Download was used, not your paths or URLs.

## References

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html): IETF. The standard: groups, user-agent matching, longest-match precedence, * and $, error handling, caching and the 500 KiB parsing limit.
- [How Google interprets the robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt): Google Search Central. The fields Google supports, the path-matching and precedence examples used above, and its handling of HTTP errors.
- [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro): Google Search Central. States that robots.txt manages crawler traffic and is not a way to keep a page out of Google.
- [List of Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers): Google Search Central. User-agent tokens for Googlebot, Googlebot-Image and Google-Extended.
- [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots): OpenAI. What GPTBot, OAI-SearchBot and ChatGPT-User are each used for.
- [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler): Anthropic. ClaudeBot, Claude-User and Claude-SearchBot, and Anthropic's support for Crawl-delay.
- [About Applebot](https://support.apple.com/en-us/119829): Apple. Applebot and Applebot-Extended, and the statement that Applebot does not follow crawl-delay.
- [To crawl or not to crawl, that is BingBot's question](https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question): Bing Webmaster Blog. Bing's 2012 post on the bingbot group and Crawl-delay values of 1 to 30.
- [do_robots()](https://developer.wordpress.org/reference/functions/do_robots/): WordPress Developer Resources. The source of the default rules behind the WordPress preset.

## Related tools

- [Meta Tags Generator](https://generate.now/meta-tags): Generate the SEO, Open Graph and Twitter Card meta tags for a page, with a link preview. An AI model can suggest the title and description from a brief.
- [UTM Link Builder](https://generate.now/utm-builder): Add utm_source, utm_medium, utm_campaign and other UTM parameters to a URL. The tool keeps existing parameters and #fragments and encodes values.
- [.gitignore Generator](https://generate.now/gitignore): Pick from 21 stack templates or describe your project in plain English. Get a labelled, deduplicated .gitignore to copy or download.
- [Slug Generator](https://generate.now/slug): Convert a title into a URL slug as you type. Choose hyphen or underscore, case and a maximum length; accents become ASCII. One title per line for bulk.

## More in Config & DevOps

- [Open Source License Generator](https://generate.now/license): Pick MIT, Apache 2.0, GPL v3, BSD, ISC, MPL 2.0, Unlicense or CC0. Get the full LICENSE file to copy or download.

[All 39 tools](https://generate.now/tools)