Every bot
that ever
knocked.
A living bestiary of 1600+ web crawlers: search by name, paste a full user-agent, see how often the top 25k sites block them.
↳ live search across patterns, descriptions, instances. Click any card.
crawlers indexed
-
categories
-
block-rate snapshot
-
most blocked today
-
summoning the bestiary…
How it works.
Crawlerdex unifies crawler detection patterns with live block-rate telemetry from its built-in radar, which scans the top 25 000 websites' robots.txt daily.
1 · Search
Type any token: gpt, ahrefs, bing, to filter crawlers by name, description, or instance string.
2 · Paste a UA
Drop a full User-Agent header into the same box. The longest-matching pattern wins; the panel reveals which crawler it is.
3 · Filter by category
Tap a chip: ai-crawler, seo, scanner, to narrow by tag.
4 · Read the graph
Each crawler's modal shows the % of top-25k sites blocking it over time. Higher line = more rejected.