Robots.txt Generator

Build a correct robots.txt from five fields: user-agent, Disallow and Allow paths, optional crawl-delay and your sitemap URL. Generated in your browser, ready to copy or download.

Every site that search engines crawl gets asked for one file before anything else: /robots.txt. It tells crawlers which URLs they may request and where your XML sitemap lives. Most sites only need a handful of lines, but the syntax is unforgiving: a stray Disallow: / hides the whole site, and a rule that is one character too short can block pages you meant to keep. This generator writes the file from five plain fields, so the format is always right and you can concentrate on which paths to block.

Use it when you launch a new site, move to a new platform, add a section crawlers should skip (cart, checkout, account pages, internal search, filtered listings), or want to point crawlers at a new sitemap. If you only need to read an existing file, the Robots.txt Tester fetches and lists the rules of any live robots.txt instead.

How to create your robots.txt

  1. User-Agent: starts as *, which means every crawler. Replace it with a single product token such as Googlebot, Bingbot or GPTBot if the rules should apply to one crawler only.
  2. Sitemap URL: the full, absolute address of your sitemap, for example https://example.com/sitemap.xml. Leave it empty if you don't have one.
  3. Disallow Paths (one per line): the URL paths crawlers should not request. Each path must start with /.
  4. Allow Paths (one per line): exceptions inside a blocked area, for example one help page inside a blocked folder.
  5. Crawl-Delay (seconds): optional. Read the note on Crawl-delay below before filling it in.
  6. Click Generate robots.txt. The file appears in the robots.txt box; use Copy to put it on the clipboard or Download to save it as robots.txt.

The generator builds the text in your browser with a few lines of JavaScript; nothing you type is sent to a server. Blank lines in the path boxes are skipped and spaces around each path are trimmed.

Worked example: an online shop

Say your store runs at example.com and you want crawlers to skip the cart, checkout, customer accounts, sorted copies of category pages and internal search, while keeping one search help page open. Enter:

  • User-Agent: *
  • Sitemap URL: https://example.com/sitemap.xml
  • Disallow Paths: /cart/, /checkout/, /my-account/, /*?orderby=, /search (one per line)
  • Allow Paths: /search/help
  • Crawl-Delay: empty

The tool outputs exactly this:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /search
Allow: /search/help

Sitemap: https://example.com/sitemap.xml

Here is how Google applies those rules to some real-looking URLs. Google picks the matching rule with the longest path, and if an Allow and a Disallow rule are equally long, Allow wins:

URL pathMatching ruleResult
/products/blue-mugnoneCrawlable
/cart/Disallow: /cart/Blocked
/shop?orderby=priceDisallow: /*?orderby=Blocked
/search?q=mugDisallow: /searchBlocked
/search/helpAllow: /search/help (longer than /search)Crawlable
/search-resultsDisallow: /searchBlocked (often a surprise)
/Cart/none (paths are case-sensitive)Crawlable

The last two rows are the classic traps. Rules are prefix matches, so /search also blocks /search-results and /searchable-guide; write /search/ or /search? if you only mean the search folder or search queries. And because matching is case-sensitive, /cart/ does not cover /Cart/.

Reading and checking the result

  • Good: every Disallow line points at pages with no search value (carts, logins, endless filter combinations), product and article URLs stay crawlable, CSS and JavaScript are not blocked, and the Sitemap line uses a full https URL.
  • Bad: Disallow: / on a live site, blocking folders that hold stylesheets, scripts or images your pages need to render, or a sitemap URL that returns 404.

Upload the file to the root of the host so it opens at https://yourdomain.com/robots.txt. A file in a subfolder is ignored, and shop.example.com needs its own file. After uploading, open it in a browser, confirm the sitemap with the Sitemap Validator, and check the robots.txt report in Google Search Console to see the version Google fetched.

Limits and edge cases

  • One group per run. The generator writes a single User-agent line with its rules. To give different crawlers different rules (for example, block GPTBot completely but let everyone else in), generate each group separately and paste them into one file with a blank line between groups. A crawler follows only the most specific group that names it, so Googlebot ignores the * group if a Googlebot group exists.
  • No syntax checking. The tool writes what you type. A path without a leading slash, a relative sitemap URL or an empty User-Agent field is output as-is, so check those fields yourself.
  • Wildcards. Google and Bing support * (any characters) and $ (end of URL), as in /*.pdf$. Some smaller crawlers don't.
  • Crawl-delay. Googlebot ignores it; Google sets its crawl rate from how your server responds. Bing and some other crawlers honour it. Only add a value if a specific bot is straining your server.
  • File size. Google reads the first 500 KiB of a robots.txt and ignores the rest.
  • Blocking is not de-indexing. A disallowed URL can still appear in Google if other pages link to it, just without a description. To keep a page out of results, let it be crawled and add a noindex robots meta tag, which you can build with the Meta Tag Generator.
  • Not security. The file is public and bad bots ignore it. Protect private areas with a login.

Compared with other approaches

Writing the file by hand in a text editor is fine if you know the rules; the generator mainly saves you from typos and forgotten colons. CMS plugins such as Yoast or Rank Math on WordPress can serve a virtual robots.txt, which is convenient but sometimes hides what is actually live, so always open the real URL to confirm. For WordPress-specific defaults, the WordPress Robots.txt Generator starts from typical WordPress paths. More examples are in our guide How to Create the Perfect Robots.txt File for SEO.

The rules described here follow the Robots Exclusion Protocol standard, RFC 9309, and Google's documentation on how Google interprets the robots.txt specification.

Frequently Asked Questions

Robots.txt rules are prefix matches, so /search matches every path that starts with those characters. Use /search/ to block only the folder, or /search? to block only search query URLs.

Google uses the matching rule with the longest path. Allow: /search/help beats Disallow: /search for that URL. If two matching rules are the same length, Google applies the Allow rule. The order of lines does not matter to Google.

The generator creates one User-agent group per run. Generate a group for each crawler (for example GPTBot and *), then paste them into one file separated by a blank line. Each crawler follows only the most specific group that names it.

No. Googlebot ignores Crawl-delay and adjusts its crawl rate based on how your server responds. Bing and some other crawlers do honour it, so only set it if one of them is overloading your server.

No. The file is assembled by JavaScript in your browser and the Download button saves it straight from the page. Nothing is uploaded or stored.