robots.txt Generator
Generate robots.txt files with user-agent rules, disallow/allow paths, and sitemap locations for your site.
Quick Presets:
Directives Config
Asterisk (*) represents all search crawlers.
Generated Output
User-agent: *
Disallow: /api/
Disallow: /admin/
Sitemap: https://joeyjazwinski.com/sitemap.xml
Frequently Asked Questions
What is robots.txt?
robots.txt is a plain text file placed in your website root directory that instructs search engine web crawlers which URLs they can or cannot request from your site.
Can robots.txt block pages from being indexed?
Robots.txt prevents crawling, but if other sites link to the URL, Google may still index it without fetching page content. To prevent indexing entirely, use the noindex meta tag instead.
Should I disallow JavaScript or CSS?
No. Googlebot and modern search crawlers need access to CSS and JavaScript to render and understand page layouts, mobile responsiveness, and dynamic content.