robots.txt & sitemap.xml checker
Verify that search engines can crawl your site: robots.txt reachability and rules, sitemap validity and URL count, Sitemap declarations — and the issues that quietly de-index pages.
The two files that control crawling
Before a search engine reads a single page, it reads robots.txt — and a single misplaced Disallow: / can remove an entire site from the index. Sitemaps are the complement: the definitive list of URLs you want discovered. Both are trivially easy to check, and both break in ways that are invisible unless you look.
This tool fetches both files exactly as a crawler would, parses the robots directives (user-agent groups, disallow/allow rules, sitemap declarations, crawl-delay), validates the sitemap XML and reports the URL count — then summarizes everything that needs attention.
Crawlability can break with any deploy. SiteFlux's SEO monitoring re-checks robots.txt, sitemap.xml and indexability around the clock and alerts you when a release accidentally blocks your pages.
Frequently asked questions
What happens if robots.txt is missing?
Nothing bad — a 404 on robots.txt is treated by Google as 'crawl everything allowed'. But you lose the ability to guide crawlers away from duplicate or low-value pages, and you miss the chance to declare your sitemap location.
How do I know if robots.txt is blocking my site?
The dangerous pattern is 'User-agent: * Disallow: /' — it forbids all crawlers from everything and de-indexes the site over time. This tool parses your robots.txt specifically looking for that rule, along with unreachable servers and parse problems.
Should my sitemap be declared in robots.txt?
Yes — a 'Sitemap: https://example.com/sitemap.xml' line is the standard way to help all search engines (not just the ones where you submitted it) discover your sitemap. The tool flags the directive if it's missing.
What makes a sitemap valid?
Well-formed XML in the urlset or sitemapindex format, with one <url> entry per page. The tool validates the XML structure, counts the URL entries, reports whether it's a sitemap index, and checks how many URLs carry a lastmod date.
Check your crawlability once — SiteFlux's SEO monitors re-verify robots.txt, sitemap and indexability after every deploy, 24/7.
Create a free monitor →