Rewrite It
Rewrite It Tool

Robots.txt Generator

Control how search engines crawl your website with this simple robots.txt generator. Generate a clean robots file with allow/disallow rules, crawl-delay, and sitemap links—then copy or download in one click.

Robots.txt Generator – Tool
Choose your crawler, set rules, add sitemap URLs, and generate a valid robots.txt file.
Pick a common bot
Optional — comma separate for multiple
One path per line (e.g., /)
One path per line (e.g., /private)
Optional; some bots ignore this
Absolute URLs — one per line

Quick options

Add any advanced directives at the end
Save as /robots.txt at your site root
Valid plain text output
Place at /robots.txt
How to Use the Robots.txt Generator
1

Pick bots & rules

Choose a user-agent (or add custom) and set allow/disallow paths that match your site structure.

2

Add sitemap & delay

Paste your sitemap URLs and set a crawl-delay if you need to slow bots during heavy traffic.

3

Copy or download

Click Copy or Download and place the file at the website root: /robots.txt.

Key Features

  • All-in-one control: Generate robots rules, add sitemap URLs, and set crawl-delay with a simple interface.

  • Presets + custom bots: Start with common crawlers (Googlebot, Bingbot, etc.) or type any custom user-agent.

  • Quick toggles: One-click options for admin folders, search pages, and URL parameters.

  • Copy & download: Get your robots file instantly as text or a ready-to-upload file.

About Robots.txt & This Robots.txt Generator

Complete your technical SEO

These tools pair perfectly with your new robots.txt.

XML Sitemap Generator

Build a fresh sitemap and include its URL in robots.txt.

Meta Tag Generator

Set page-level instructions like noindex and social previews.