Robots.txt Generator
Create a robots.txt rule file to guide search engine bots.
What Is robots.txt?
The robots.txt file is a plain text file placed at your website's root (e.g., example.com/robots.txt) that tells search engine crawlers which pages or directories they should or shouldn't access. It's part of the Robots Exclusion Protocol (REP) — a standard followed by all major search engines.
Common robots.txt Directives
- User-agent: * — applies rules to all bots
- Disallow: /admin/ — blocks crawling of /admin/ directory
- Allow: / — explicitly allows crawling (overrides Disallow for specific paths)
- Sitemap: https://example.com/sitemap.xml — tells bots where your sitemap is
- Crawl-delay: 10 — asks bots to wait 10 seconds between requests (not supported by Google)
What NOT to Block with robots.txt
- CSS and JS files — Google needs these to render and understand your pages
- Pages you want indexed but need auth to view — use noindex instead
- Image files used on important pages — Google Images won't index them
Does robots.txt prevent a page from being indexed?
Not reliably. Robots.txt prevents crawling, but Google can still index a URL it has seen linked from other pages even if it can't crawl it. The URL may appear in search results without a snippet ("A description for this result is not available"). To definitively block indexing, use the noindex meta tag or X-Robots-Tag HTTP header on the page itself.
Can I block specific bots like scrapers?
You can specify rules for specific bots by user-agent (e.g., User-agent: GPTBot to block OpenAI's crawler). However, robots.txt is voluntary — legitimate search bots follow it, but malicious scrapers and bad actors typically ignore it entirely. For effective protection against scrapers, use IP-based blocking, rate limiting, or CAPTCHA at the server/CDN level.