Free tool · No signup
Robots.txt & Sitemap Tester
Discover technical SEO and crawler-access problems on any public website: is robots.txt reachable and valid, which sitemaps does it declare, are those sitemaps valid XML, and do they actually list your important pages? This is a limited technical check — not a full SEO audit.
What this check covers
- Fetches /robots.txt and shows the HTTP status and syntactic validity.
- Extracts Sitemap: directives and inspects each sitemap URL.
- Checks /sitemap.xml plus sitemaps declared in robots.txt (up to 6 documents).
- Validates basic XML structure, URL syntax, and the sitemaps.org namespace (external entities rejected).
- Reports sitemap URL counts (up to 5,000 URLs evaluated per document).
- Tests up to 10 important URLs for sitemap inclusion.
- Explains robots.txt directives for the relevant crawler groups per path — a simplified reading of the file, not a full robots-protocol implementation.
Limits of this tool
- It checks technical accessibility only — not content quality, rankings, backlinks, or penalties.
- A passing result does not guarantee indexing; search engines decide what to index.
- Only public websites can be checked (no localhost, private IPs, or login-walled hosts).
- Results reflect the moment of the check — re-run after you deploy fixes (crawlers also cache robots.txt).
Keep checking crawler readiness
Pair this with the agent-readiness audit and the agent endpoint checker.