TL-37 · Free · No signup

Robots.txt Generator

Generate a robots.txt file that tells search engine crawlers which parts of your site they can and can't access.

TL-37

Generate a robots.txt file that tells search engine crawlers which parts of your site they can and can't access.

Default rule

Paths to disallow (one per line)

Sitemap URL (optional)

Output

How to use the Robots.txt Generator

robots.txt controls crawling, not indexing — a blocked page can still appear in search results if it's linked from elsewhere.
Place the file at the root of your domain, e.g. example.com/robots.txt, for it to be recognized.