Robots.txt Tester

Free Robots.txt Tester — validate your crawl directives instantly

The Robots.txt Tester is a free robots.txt checker and robots.txt validator that fetches the /robots.txt file at your site’s origin and parses it into clear, readable rules. Enter any domain and this tool retrieves the live file, groups every Allow and Disallow rule by User-agent, surfaces any Crawl-delay values, and lists every Sitemap: URL it declares. If your site has no robots.txt and the request returns a 404, the tester reports that plainly — so you always know exactly what search engine crawlers see when they visit your crawl directives.

How the Robots.txt Tester works

Checking your robots.txt takes three simple steps:

Enter your domain. Paste any website address into the box above and run the test.
We fetch and parse the file. The tool requests /robots.txt at the site’s origin and reads every directive directly from the source — nothing is guessed or estimated.
You get a structured report. Rules are grouped per user-agent with their Allow, Disallow and Crawl-delay lines, and any declared sitemap URLs are listed so you can confirm your crawlability setup at a glance.

What is robots.txt and where does it live?

robots.txt is a plain-text file that follows the Robots Exclusion Protocol. It tells search engine crawlers — such as Googlebot and Bingbot — which parts of your site they may or may not request. The file must sit at the root of your origin, for example https://example.com/robots.txt; crawlers look for it there and nowhere else. Because it is scoped to a single origin, your www and non-www hosts, and your http and https versions, each need their own file. It is one of the first things a crawler fetches, which makes it a cornerstone of technical SEO.

How User-agent, Allow, Disallow and Crawl-delay work

A robots.txt file is made up of one or more groups, each beginning with a User-agent line that names the crawler the rules apply to. A wildcard User-agent: * targets every bot, while User-agent: Googlebot targets one specifically. Inside each group you place path rules.

Disallow

Disallow asks the matched crawler not to fetch URLs beginning with the given path. For example Disallow: /admin/ blocks the admin area, while Disallow: / blocks the entire site. An empty Disallow: value means nothing is blocked.

Allow

Allow creates an exception inside a disallowed path. For instance, Disallow: /files/ combined with Allow: /files/public/ blocks the folder while still permitting one subfolder. Most crawlers resolve conflicts by matching the most specific rule.

Crawl-delay

Crawl-delay requests a pause, in seconds, between successive requests from a crawler. Support varies — Bing and some others honour it, while Googlebot ignores it in favour of its own crawl-rate controls. Our tester still reports any Crawl-delay value it finds so you have a complete picture of your directives.

Declaring your XML sitemap in robots.txt

You can point crawlers to your XML sitemap by adding one or more Sitemap: lines anywhere in the file, each with the full absolute URL — for example Sitemap: https://example.com/sitemap.xml. These lines are independent of any User-agent group and apply to every crawler, giving search engines a reliable way to discover all of your important pages. The robots.txt tester lists each sitemap URL it finds so you can verify the declaration is present and points to a valid file.

Robots.txt vs meta robots vs noindex

This distinction matters more than any other on this page: Disallow blocks crawling, not indexing. When you Disallow a URL, you are only asking crawlers not to fetch it — you are not asking Google to remove it from search results. If other pages link to that blocked URL, Google may still index it (often showing a bare URL with no description) precisely because it was never allowed to crawl the page and read any instructions on it.

To actually keep a page out of the index, do the opposite of blocking: allow crawling and add a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header. The crawler must be able to fetch the page to see that noindex — which is why a Disallow and a noindex on the same URL work against each other. Use robots.txt to manage crawling, and meta robots to manage indexing.

Crawl-budget considerations

On large sites, robots.txt is a practical crawl-budget lever. By disallowing low-value paths — faceted-search parameters, infinite filter combinations, internal search results, staging endpoints — you steer crawlers away from crawl traps and toward the pages that actually matter. That said, most small and mid-sized sites do not need aggressive blocking; over-restricting crawlability is a bigger risk than letting a few thin pages be crawled. Test changes here carefully before deploying them.

Robots.txt best-practices checklist

Serve the file at the origin root, https://yourdomain.com/robots.txt, with a 200 response.
Keep it valid UTF-8 plain text; do not return HTML or an error page in its place.
Group rules clearly under each User-agent, using * for the default group.
Never Disallow your CSS or JS if it prevents Google from rendering the page correctly.
Declare at least one Sitemap: line with an absolute URL.
Use meta robots noindex — not Disallow — to remove pages from search results.
Re-test with a robots.txt validator after every change to confirm intent.

Common robots.txt mistakes

Accidentally shipping Disallow: / from a staging config, blocking the entire site.
Expecting Disallow to remove a page from Google when it only blocks crawling.
Placing the file in a subfolder instead of the origin root, so crawlers never find it.
Blocking CSS and JavaScript, breaking how Google renders and evaluates the page.
Using relative sitemap paths instead of full absolute URLs.
Forgetting that each subdomain and protocol needs its own robots.txt file.

Ready to build or fix your file? Use our free Robots.txt Generator to produce clean, valid User-agent, Disallow, Allow and Sitemap lines, then re-run this robots.txt tester to confirm your crawl directives parse exactly as intended. You can also run a full Website SEO Checker report to audit titles, headings, links, structured data and crawlability in one place.

Frequently asked questions

What is a robots.txt tester?

A robots.txt tester is a free technical SEO tool that fetches the /robots.txt file at your site’s origin and parses it into readable rules. It shows the Allow and Disallow directives grouped by each User-agent, lists any Crawl-delay values, and collects every Sitemap: URL declared in the file. If the file returns a 404, the tester reports that as a fact so you know no robots.txt exists. Everything shown is observed directly from your live file — nothing is guessed.

Does Disallow in robots.txt stop a page from being indexed?

No — this is the most common misunderstanding. A Disallow rule asks crawlers not to fetch a URL, but it does not remove that URL from Google’s index. If other pages link to the blocked URL, Google can still list it in search results (often with no description) because it never crawled the page to see a noindex tag. To keep a page out of search results, allow crawling and use a meta robots noindex tag or an X-Robots-Tag HTTP header instead.

Where should the robots.txt file be located?

The robots.txt file must live at the root of your origin, for example https://example.com/robots.txt. Crawlers only look for it there — a file placed in a subfolder such as /blog/robots.txt is ignored. Each subdomain and each protocol needs its own file, so www and non-www, and http and https, are treated separately.

Can I declare my XML sitemap in robots.txt?

Yes. You can add one or more Sitemap: lines anywhere in robots.txt, each pointing to the full absolute URL of an XML sitemap or sitemap index. This is independent of User-agent groups and gives every crawler an easy way to discover your sitemap. Our robots.txt checker lists each Sitemap URL it finds so you can confirm it is present and correct.

Is the robots.txt tester free?

Yes, it is completely free with no signup. Enter any domain and the tool instantly fetches and validates the live robots.txt, showing per-user-agent rules, crawl-delay and sitemap declarations. Every result is read directly from the file — no data is invented or estimated.