Robots.txt Generator

Free Robots.txt Generator — create a valid robots.txt file in seconds

The Robots.txt Generator is a free, client-side tool that helps you create a robots.txt file without memorising the syntax. Choose whether to allow or disallow crawling by default, add the paths you want to block, set an optional Crawl-delay, and drop in your Sitemap URL. The tool instantly builds a clean, standards-compliant robots.txt file you can copy and paste straight to your server. It all runs in your browser — no data is ever sent to a server — making it a safe, private way to handle this piece of technical SEO.

How the robots.txt generator works

Building your file takes three simple steps:

1. Choose your default policy. Pick Allow all to let crawlers access your whole site, or Disallow all to block everything by default and open up only what you need.
2. Add your rules and sitemap. Enter your Disallow paths (one per line), add an optional Crawl-delay, and paste your sitemap URL so search engines can find it.
3. Copy the output. The generator produces a ready-to-use robots.txt file. Copy it, save it as robots.txt, and upload it to your site root.

What is a robots.txt file and how do you use it?

A robots.txt file is a plain-text file that tells search engine crawlers which parts of your site they may or may not request. It follows the Robots Exclusion Protocol and is the first file most bots look for when they visit a domain. You use it to keep crawlers out of admin areas, staging paths, internal search results, duplicate URLs and other pages that add no value to search — helping bots spend their crawl budget on the content that matters. Because it is advisory, well-behaved crawlers such as Googlebot and Bingbot obey it, but it is not a security barrier and will not hide private data.

The robots.txt syntax this tool generates

Every rule this generator writes uses a small, well-defined set of directives:

User-agent

The User-agent line names the crawler a block of rules applies to. User-agent: * targets all bots, while a specific value such as User-agent: Googlebot targets one crawler. This tool starts each file with a User-agent line so your rules always have a clear scope.

Allow and Disallow

Disallow tells crawlers not to fetch URLs that begin with the given path, and Allow creates an exception inside a blocked area. For example Disallow: /admin/ blocks the admin folder, while Disallow: (empty) or Allow: / permits everything. The allow / disallow pairing is the core of any robots.txt file.

Crawl-delay

Crawl-delay asks supported crawlers to wait a set number of seconds between requests to ease server load. It is optional, and Google ignores it, so only add it when you genuinely need to slow bots down.

Sitemap

The Sitemap directive points crawlers to your XML sitemap with an absolute URL, for example Sitemap: https://example.com/sitemap.xml. Adding it helps search engines discover all of your important pages faster.

How to add and upload your robots.txt file

Once you make your robots.txt with the generator, save the output in a file named exactly robots.txt (all lowercase). Upload it to the root directory of your domain so it is reachable at example.com/robots.txt — a robots.txt file only works at the site root, never in a subfolder. Use FTP, your hosting control panel's file manager, or your CMS to place the file, then open the URL in your browser to confirm it loads correctly. If you run multiple subdomains, each subdomain needs its own robots.txt at its own root.

Robots.txt vs meta robots and noindex

It is easy to confuse the two, but they do different jobs. A Disallow rule in robots.txt controls crawling — it stops bots from fetching a URL — but it does not control indexing. A URL you block can still appear in search results if other pages link to it, because Google never crawls it to see a noindex instruction. If your goal is to keep a page out of the index, do the opposite: let it be crawled and add a meta robots noindex tag (or an X-Robots-Tag HTTP header). Use robots.txt to manage crawl budget, and meta robots noindex to control what actually shows up in search.

Robots.txt best-practices checklist

• Place the file at your site root so it resolves at example.com/robots.txt.
• Start with a User-agent line, then list your allow / disallow rules.
• Only block paths that add no search value — admin, cart, internal search, staging.
• Add your Sitemap URL as an absolute link so bots find your XML sitemap.
• Keep the file small and readable; comment lines start with #.
• Use meta robots noindex, not Disallow, when you want a page kept out of the index.
• Test the file after uploading to be sure the right pages are blocked and allowed.

Common robots.txt mistakes to avoid

• Blocking CSS and JavaScript. Disallowing your assets stops Google from rendering pages properly and can hurt rankings.
• Disallowing the whole site. A stray Disallow: / under User-agent: * can deindex your entire website.
• Expecting Disallow to hide pages. Blocked URLs can still be indexed via external links — use noindex instead.
• Using a relative sitemap URL. The Sitemap directive must be an absolute, full URL.
• Placing the file in a subfolder. Only /robots.txt at the root is read by crawlers.
• Leaving a staging block live. A test-time Disallow: / pushed to production is a common traffic-loss disaster.

Once you have generated your file, verify it works with our free Robots.txt Tester to confirm the right URLs are allowed and blocked. Then run a full Website SEO Checker report to audit your titles, headings, links, sitemap and other technical SEO signals in one place.

Frequently asked questions

What is a robots.txt generator?

A robots.txt generator is a free tool that builds a valid robots.txt file for you without writing the syntax by hand. You choose whether to allow or disallow crawling by default, add any Disallow paths you want to block, set an optional Crawl-delay, and add your Sitemap URL. The generator then outputs a clean, copy-paste-ready robots.txt file. This tool runs entirely in your browser, so nothing you type is ever sent to a server.

Where do I put the robots.txt file?

A robots.txt file must live in the root directory of your domain so search engines can find it at yourdomain.com/robots.txt (for example, example.com/robots.txt). It will not work in a subfolder such as example.com/blog/robots.txt. Upload it to the root of your web server via FTP, your hosting file manager, or your CMS, then visit the URL in a browser to confirm it loads.

Does Disallow in robots.txt stop a page from being indexed?

No. A Disallow rule stops compliant crawlers from fetching a URL, but it does not guarantee the page is removed from search results. If other sites link to a blocked URL, Google can still list it without a description. To keep a page out of the index, allow it to be crawled and add a meta robots noindex tag (or an X-Robots-Tag header) instead of blocking it in robots.txt.

What is Crawl-delay in robots.txt?

Crawl-delay tells some crawlers how many seconds to wait between requests, which can reduce load on a small or slow server. It is optional and not part of the original standard. Googlebot ignores Crawl-delay (you set the crawl rate in Search Console instead), but crawlers such as Bing and Yandex do respect it. Only add a Crawl-delay if you actually need to throttle bots.

Is this robots.txt generator free and private?

Yes. The generator is completely free with no signup, and it works entirely client-side in your browser. Your rules, paths and sitemap URL are never uploaded or stored anywhere — the robots.txt file is built locally on your device and you simply copy the result.