What is a Robots.txt File?
A robots.txt file is a text file placed in your website's root directory that tells search engine crawlers which pages or sections of your site they should or shouldn't visit.
Why Your Website Needs a Robots.txt File
- Crawl budget optimization — Direct search engines to your most important pages
- Prevent indexing of duplicate content
- Protect sensitive areas
- Sitemap discovery
Basic Robots.txt Syntax
- User-agent: — Specifies which crawler the rules apply to
- Disallow: — Blocks access to specific paths
- Allow: — Explicitly allows access
- Sitemap: — Specifies the location of your XML sitemap
Creating Your Robots.txt File
The easiest way to create a robots.txt file is using our Robots.txt Generator.
Common Mistakes to Avoid
- Blocking CSS and JavaScript files — Google needs these to render your pages
- Using robots.txt for security — It's publicly accessible
- Blocking your entire site — A misplaced "Disallow: /" can de-index your whole website
- Forgetting the sitemap directive
- Not testing changes