Free robots.txt checker
Paste a site URL (or a full robots.txt URL). We fetch the file, show the rules, and let you test any path to see whether a crawler is allowed orblocked. No signup, no credit card.
Enter a URL to inspect robots.txt…
How robots.txt rules work
robots.txt tells crawlers which paths are off-limits. The most specific matching rule wins, and Allow beats Disallow at the same specificity. This checker parses the rules the way major search engines do and lets you test any path against them.
Ensure your site is crawlable
Fastcrawl's map endpoint follows your robots rules and reports every crawlable URL on a site. Free tier: 2,000 credits, no card.
Check your whole site's crawlability. 2,000 free credits, no card.
Get a free API key → · Read the docs