Allmystery UrlBot
URL verification bot
About this crawler
Allmystery UrlBot is a web crawler identified by the regular-expression pattern Allmystery UrlBot in the User-Agent request header. It is categorised as monitoring. Use the regex above to detect, log, allow, or block Allmystery UrlBot traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
No block-rate data for this crawler.
Technical details
- Name
- Allmystery UrlBot
- Pattern
Allmystery UrlBot- Tags
- monitoring
- Reference
- https://www.webrobots.de/bot/urlbot/
- Added
- 2026/05/03
- Instances
- 1 known sample(s)
Sample User-Agent strings
Allmystery UrlBot/1.3
Block this crawler
robots.txt — disallow Allmystery UrlBot:
User-agent: Allmystery UrlBot
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Allmystery UrlBot [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "Allmystery UrlBot") {
return 403;
}
← back to all crawlers