Build a correct robots.txt from five fields: user-agent, Disallow and Allow paths, optional crawl-delay and your sitemap URL. Generated in your browser, ready to copy or download.
Every site that search engines crawl gets asked for one file before anything else: /robots.txt. It tells crawlers which URLs they may request and where your XML sitemap lives. Most sites only need a handful of lines, but the syntax is unforgiving: a stray Disallow: / hides the whole site, and a rule that is one character too short can block pages you meant to keep. This generator writes the file from five plain fields, so the format is always right and you can concentrate on which paths to block.
Use it when you launch a new site, move to a new platform, add a section crawlers should skip (cart, checkout, account pages, internal search, filtered listings), or want to point crawlers at a new sitemap. If you only need to read an existing file, the Robots.txt Tester fetches and lists the rules of any live robots.txt instead.
*, which means every crawler. Replace it with a single product token such as Googlebot, Bingbot or GPTBot if the rules should apply to one crawler only.https://example.com/sitemap.xml. Leave it empty if you don't have one./.robots.txt.The generator builds the text in your browser with a few lines of JavaScript; nothing you type is sent to a server. Blank lines in the path boxes are skipped and spaces around each path are trimmed.
Say your store runs at example.com and you want crawlers to skip the cart, checkout, customer accounts, sorted copies of category pages and internal search, while keeping one search help page open. Enter:
*https://example.com/sitemap.xml/cart/, /checkout/, /my-account/, /*?orderby=, /search (one per line)/search/helpThe tool outputs exactly this:
User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /*?orderby= Disallow: /search Allow: /search/help Sitemap: https://example.com/sitemap.xml
Here is how Google applies those rules to some real-looking URLs. Google picks the matching rule with the longest path, and if an Allow and a Disallow rule are equally long, Allow wins:
| URL path | Matching rule | Result |
|---|---|---|
| /products/blue-mug | none | Crawlable |
| /cart/ | Disallow: /cart/ | Blocked |
| /shop?orderby=price | Disallow: /*?orderby= | Blocked |
| /search?q=mug | Disallow: /search | Blocked |
| /search/help | Allow: /search/help (longer than /search) | Crawlable |
| /search-results | Disallow: /search | Blocked (often a surprise) |
| /Cart/ | none (paths are case-sensitive) | Crawlable |
The last two rows are the classic traps. Rules are prefix matches, so /search also blocks /search-results and /searchable-guide; write /search/ or /search? if you only mean the search folder or search queries. And because matching is case-sensitive, /cart/ does not cover /Cart/.
Disallow: / on a live site, blocking folders that hold stylesheets, scripts or images your pages need to render, or a sitemap URL that returns 404.Upload the file to the root of the host so it opens at https://yourdomain.com/robots.txt. A file in a subfolder is ignored, and shop.example.com needs its own file. After uploading, open it in a browser, confirm the sitemap with the Sitemap Validator, and check the robots.txt report in Google Search Console to see the version Google fetched.
User-agent line with its rules. To give different crawlers different rules (for example, block GPTBot completely but let everyone else in), generate each group separately and paste them into one file with a blank line between groups. A crawler follows only the most specific group that names it, so Googlebot ignores the * group if a Googlebot group exists.* (any characters) and $ (end of URL), as in /*.pdf$. Some smaller crawlers don't.noindex robots meta tag, which you can build with the Meta Tag Generator.Writing the file by hand in a text editor is fine if you know the rules; the generator mainly saves you from typos and forgotten colons. CMS plugins such as Yoast or Rank Math on WordPress can serve a virtual robots.txt, which is convenient but sometimes hides what is actually live, so always open the real URL to confirm. For WordPress-specific defaults, the WordPress Robots.txt Generator starts from typical WordPress paths. More examples are in our guide How to Create the Perfect Robots.txt File for SEO.
The rules described here follow the Robots Exclusion Protocol standard, RFC 9309, and Google's documentation on how Google interprets the robots.txt specification.