Robots.txt Generator

Build a robots.txt file with Allow and Disallow rules per crawler, sitemap lines and presets for blocking AI training bots. Copy or download it.

Start from a preset (replaces the rules below)

robots.txt · 5 lines · 96 B

User-agent: *
Disallow: /admin/
Allow: /admin/public/

Sitemap: https://example.com/sitemap.xml

What this file does

Every crawler may crawl everything except URLs starting with /admin/. /admin/public/ stays allowed because the longer matching rule wins.
One sitemap is listed for any crawler that reads the Sitemap line.

robots.txt is a request, not access control: a crawler can ignore it, and a disallowed URL can still be indexed if other pages link to it. Upload the file to the root of the host it applies to, as /robots.txt.

Group 1

Applies to (User-agent)

Rules

Paths start with / and are case-sensitive. * matches any characters and $ marks the end of the URL.

Crawl-delay in seconds (optional)

Not part of RFC 9309. Google ignores it; Bing accepts 1 to 30.

Sitemaps

Enter full URLs. Sitemap lines apply to the whole file, not to one group.

Check a URL against these rules

The check uses the matching in RFC 9309: the longest matching path wins, Allow wins a tie, and paths are case-sensitive.

Crawler

URL path

Allowed. No group names Googlebot, so the * group applies. Allow: /admin/public/ is the longest match, ahead of Disallow: /admin/.

The file is built in your browser. Your rules are not sent to our server.

01 · Definition

What is a robots.txt file?

robots.txt is a plain text file at the root of a website that tells crawlers which URLs they may request. It is defined by RFC 9309 and made of groups: one or more User-agent lines followed by Allow and Disallow rules. It is a request to crawlers, not access control, and it does not remove pages from search results.

02 · Overview

How the Robots.txt Generator works

The tool builds a robots.txt from rule groups: pick the crawlers a group applies to, add Allow and Disallow paths, and list your sitemaps. Presets cover allowing everything, blocking everything, blocking the crawlers whose operators say they collect data for AI training, and the rules WordPress serves by default. Paths, user-agent tokens and sitemap URLs are validated as you type. A plain-English summary explains what the file does, and a checker tells you whether a given URL is allowed for a given crawler using the longest-match rule from RFC 9309. The file is built in your browser and is not sent to our server.

03 · Guide

How crawlers decide which rule applies

A crawler first looks for a group that names its own token, compared without regard to case. If one exists it follows only that group and ignores the * group. If none does, it follows the * group. If there is no * group either, nothing is restricted.

Inside the group, every Allow and Disallow path is compared with the start of the URL path, and paths are case-sensitive. The rule with the longest matching path wins. If an Allow and a Disallow match with the same length, Allow wins. * matches any run of characters and $ marks the end of the URL. The examples are from Google's documentation and give the same result in this tool's checker.

Rules	URL path	Result	Why
Allow: /p and Disallow: /	/page	Allowed	/p is the longer match
Allow: /folder and Disallow: /folder	/folder/page	Allowed	Same length, Allow wins
Allow: /page and Disallow: /*.htm	/page.htm	Disallowed	/*.htm is the longer match
Allow: /$ and Disallow: /	/page.htm	Disallowed	/$ matches only the home page
Disallow: /fish	/Fish.asp	Allowed	Paths are case-sensitive

Which rule wins

04 · Guide

AI crawler user agents and what each one is for

Each row is taken from the operator's own documentation, read on 11 November 2026. The "Block AI training" preset disallows the six tokens marked Yes: the ones whose operators say the crawl may be used to train models. It leaves AI search crawlers and user-requested fetches alone, so a site can still be found and cited in AI answers.

Google-Extended and Applebot-Extended are not separate crawlers. They are tokens that control how pages already fetched by Googlebot and Applebot may be used, and both companies say that disallowing them does not affect search results. OpenAI and Perplexity say robots.txt may not apply to fetches a user asks for.

Token	Operator	Used for	In preset
GPTBot	OpenAI	Content that may train models	Yes
ClaudeBot	Anthropic	Content that may train models	Yes
Google-Extended	Google	Use of crawled pages for Gemini training	Yes
Applebot-Extended	Apple	Use of crawled pages for Apple's models	Yes
Meta-ExternalAgent	Meta	Training AI models, indexing	Yes
Amazonbot	Amazon	Amazon products; may train models	Yes
OAI-SearchBot	OpenAI	ChatGPT search results	No
ChatGPT-User	OpenAI	Pages a user asks ChatGPT to open	No
Claude-SearchBot	Anthropic	Claude search results	No
Claude-User	Anthropic	Pages a user asks Claude to open	No
PerplexityBot	Perplexity	Perplexity search results	No

AI user-agent tokens

05 · Use cases

When to use the Robots.txt Generator

01
Write the first robots.txt for a new site, with a Sitemap line
02
Keep crawlers out of admin, cart and internal search URLs
03
Opt out of AI training crawlers while staying visible in search
04
Check whether a URL is allowed for a crawler before deploying a rule change

06 · Examples

Robots.txt Generator examples

Preset: Allow all

ex 01

User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml

An empty Disallow matches nothing, so every crawler may fetch every URL. Replace the sitemap URL with your own.

Preset: Block AI training

ex 02

User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: Amazonbot Disallow: / User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml

One group disallows the whole site for six tokens. Every other crawler falls through to the * group, which allows everything.

Preset: WordPress

ex 03

User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/wp-sitemap.xml

The rules WordPress core generates. admin-ajax.php stays crawlable because its Allow path is longer than the Disallow path.

Check /admin/public/report.html for Googlebot against Disallow: /admin/ and Allow: /admin/public/

ex 04

Allowed. No group names Googlebot, so the * group applies. Allow: /admin/public/ is the longest match, ahead of Disallow: /admin/.

The checker names the group that applied and the rule that won.

07 · FAQ

Robots.txt: frequently asked questions

Does robots.txt remove a page from Google?

No. Google's documentation says robots.txt "is not a mechanism for keeping a web page out of Google". A disallowed URL can still be indexed if other pages link to it, and then appears without a description. To keep a page out of search results, let it be crawled and add a noindex robots meta tag or header, or put it behind a password.

Where does the robots.txt file go?

It goes at the top level of the host, named exactly /robots.txt and served as UTF-8 plain text. It applies only to the protocol, host and port it is served from, so https://example.com/robots.txt does not cover shop.example.com or the http version. A file in a subdirectory is not read.

Which rule wins when Allow and Disallow both match?

The one with the longest path. RFC 9309 says the most specific match, the one with the most octets, must be used, and that Allow should win when the two are equal. Order in the file does not matter. With Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php, admin-ajax.php is allowed and the rest of /wp-admin/ is not.

Do crawlers with their own group also follow the * rules?

No. A crawler that finds a group naming it uses only that group. If Googlebot has its own group and you also want it kept out of /admin/, repeat Disallow: /admin/ there. The summary under the output points this out whenever your file has both named groups and a * group.

Does Google obey Crawl-delay?

No. Crawl-delay is not part of RFC 9309, and Google's documentation lists it as a field it does not support. Bing has said it honours values from 1 to 30, Anthropic says ClaudeBot supports it, and Apple and Amazon say their crawlers do not follow it. The tool writes the line only if you fill the field in.

How do I block AI crawlers with robots.txt?

Add a group that names their tokens and disallows /. The "Block AI training" preset does this for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and Amazonbot, the tokens whose operators say the crawl may be used for model training. robots.txt is a request: it works for crawlers that choose to honour it and does nothing against one that ignores it.

What happens if robots.txt is missing or returns an error?

Under RFC 9309, a 4xx response such as 404 means the file is unavailable and the crawler may access everything. A 5xx response means it is unreachable and the crawler must assume the whole site is disallowed. Crawlers should not use a cached copy for more than 24 hours, and must be able to parse at least 500 kibibytes; Google ignores anything past that size.

Are my rules sent to your server?

No. The file, the summary and the URL check are all computed in the page. The page loads standard page analytics, which record the visit and that a preset, Copy or Download was used, not your paths or URLs.

08 · Reference

Specs and further reading

These are the primary sources this tool follows. Where a specification defines the behaviour, we link the specification, not a summary of it.

RFC 9309: Robots Exclusion Protocol
IETF. The standard: groups, user-agent matching, longest-match precedence, * and $, error handling, caching and the 500 KiB parsing limit.
How Google interprets the robots.txt specification
Google Search Central. The fields Google supports, the path-matching and precedence examples used above, and its handling of HTTP errors.
Introduction to robots.txt
Google Search Central. States that robots.txt manages crawler traffic and is not a way to keep a page out of Google.
List of Google's common crawlers
Google Search Central. User-agent tokens for Googlebot, Googlebot-Image and Google-Extended.
Overview of OpenAI Crawlers
OpenAI. What GPTBot, OAI-SearchBot and ChatGPT-User are each used for.
Does Anthropic crawl data from the web, and how can site owners block the crawler?
Anthropic. ClaudeBot, Claude-User and Claude-SearchBot, and Anthropic's support for Crawl-delay.
About Applebot
Apple. Applebot and Applebot-Extended, and the statement that Applebot does not follow crawl-delay.
To crawl or not to crawl, that is BingBot's question
Bing Webmaster Blog. Bing's 2012 post on the bingbot group and Crawl-delay values of 1 to 30.
do_robots()
WordPress Developer Resources. The source of the default rules behind the WordPress preset.

09 · More

Tools that pair with Robots.txt

Meta Tags Generator

Generate the SEO, Open Graph and Twitter Card meta tags for a page, with a link preview. An AI model can suggest the title and description from a brief.

Web/meta-tags

UTM Link Builder

Add utm_source, utm_medium, utm_campaign and other UTM parameters to a URL. The tool keeps existing parameters and #fragments and encodes values.

Web/utm-builder

.gitignore Generator

Pick from 21 stack templates or describe your project in plain English. Get a labelled, deduplicated .gitignore to copy or download.

Config/gitignore

Slug Generator

Convert a title into a URL slug as you type. Choose hyphen or underscore, case and a maximum length; accents become ASCII. One title per line for bulk.

Text/slug