Robots.txt Generator
Generate robots.txt file for your website. Control which pages search engines can crawl. Free online robots.txt generator tool.
0 = no delay. Recommended: 10 for shared hosting
📖 Robots.txt Guide
What is robots.txt?
A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site.
| User-agent: | Specifies the crawler |
| Disallow: | Blocks access to path |
| Allow: | Grants access to path |
| Sitemap: | URL of your XML sitemap |
| Crawl-delay: | Seconds between requests |
📋 Common Rules
- /wp-admin/ - Block WordPress admin
- /wp-includes/ - Block WP core files
- /*.pdf$ - Block PDF files
- /temp/ - Block temporary files
- /cgi-bin/ - Block scripts
📝 Quick Templates
How to Generate robots.txt
Create your robots.txt file in 3 simple steps
1
Select Crawler
Choose which search engine bot to set rules for.
2
Add Rules
Add disallow/allow paths and sitemap URL.
3
Download
Copy or download the generated robots.txt file.
Related SEO Tools
Explore more tools in the seo tools category