Set user-agent groups and path rules, then review the generated file.
The robots.txt Generator helps you create a robots.txt file with user-agent groups, path rules, and an optional sitemap URL. Review the output before placing it on your website.
A robots.txt file gives instructions to crawlers that choose to follow them. It does not enforce authentication or hide sensitive paths.
A Disallow rule concerns crawling, not a guarantee against indexing. To ask a search engine not to index a page, a noindex directive must be accessible to that crawler. Do not block the fetch that it needs to read the directive.
This example disallows an admin path, allows other paths, and supplies a sitemap location:
User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xmlThe rule for /admin/ is a crawler instruction, not protection for that area. Use authentication for private content.
Be careful with Disallow: /: it asks the matching crawler not to fetch the whole site. Check the relevant group and path rules rather than relying on a general allow rule elsewhere in the file.
Publish the file at the site root, as in example.com/robots.txt, and verify the served content. If you use noindex on a page, check that the crawler can still access it. The interface does not show a crawl-delay control.
Use the .htaccess Generator for Apache redirects and other server rules. Those settings have a different purpose from crawler instructions.
How to Use robots.txt Generator - Create Crawler Rules
Enter a User-Agent and choose Custom paths, Allow all, or Disallow all. For custom rules, enter one path per line in the relevant fields.
Enter your existing sitemap URL in Sitemap URL. Use Add user-agent group if another crawler needs separate rules.
Check Generated robots.txt, then use Copy or Download robots.txt. Publish it at your site root and verify that the intended paths remain crawlable.
Set user-agent groups and path rules, then review the generated file.
Tool Use:
6Added:
Sep 11, 2026Type:
Free ToolPrivacy:
Client SideQR code generator for URLs and text. Adjust dimensions, margin, colors, and an optional logo, then download the code and test it before sharing.
Stylish text generator with fonts, colors, gradients, shadows, and strokes. Set the canvas size, preview your design, and download it as a PNG.
Random data generator for test datasets and prototypes. Choose fields and a row count, generate sample records, and review the output before exporting.