Robots.txt is the first file every crawler reads. Sitemap.xml is the map crawlers use to find what to read next. Get either wrong and Google may quietly ignore huge parts of your site.
This tool fetches both files and shows you what a crawler would see. It flags the most common killer: a Disallow: / rule that blocks the whole site.
Where this tool actually helps
Post-launch check
New sites often ship with Disallow: /. Run this within a day of launch.
After a redesign
CMS migrations often rewrite robots.txt. A one-minute check saves weeks of lost traffic.
Client onboarding
First file we always look at on a new client. Tells you what state their SEO is really in.
How to read the result
If robots.txt is missing, most crawlers assume the site is fully open. That is fine, but you lose the chance to point them at your sitemap.
A sitemap with zero URLs, or a 404, is a red flag. Google will still crawl your site, but new URLs will be discovered slowly. Fix the sitemap and Search Console indexing catches up within days.
Common mistakes we see
- Leaving Disallow: / in production after launch.
- Referencing a sitemap URL that returns 404 in robots.txt.
- Splitting a large site into a sitemap index but forgetting to list every sub-sitemap.
- Blocking /wp-content or /assets, then wondering why images do not rank.
FAQs
- Do I need a robots.txt?
- It is optional, but useful. It tells crawlers where your sitemap is and lets you block low-value pages like admin or search results.
- Where should I put my sitemap?
- At /sitemap.xml on your root domain, and referenced in robots.txt. Submit it in Google Search Console too.
- What if my sitemap has thousands of URLs?
- Split it into multiple sitemaps and use a sitemap index. Each sitemap should stay under 50,000 URLs and 50MB.
- Does Disallow prevent indexing?
- No. It prevents crawling. To keep a page out of the index, use a noindex meta tag and allow crawling so Google can see the tag.
- Do you store the URL?
- No. It is fetched once for the check and discarded.
A note from Nimitt
A tool gives you a number. A person tells you what to do with it. If you want a straight answer on your site, send it over. I read every one myself.