crawl diagnostics

Robots & Sitemap Checker

Fetch robots.txt, detect crawler blocks, discover sitemap URLs, and parse sitemap XML.

Safety note This checker only fetches public HTTP/HTTPS URLs and blocks local/private network addresses.