Xenu Link Sleuth
Xenu link checker web crawler tool
About this crawler
Xenu Link Sleuth is a web crawler identified by the regular-expression pattern Xenu Link Sleuth in the User-Agent request header. It is categorised as seo. Use the regex above to detect, log, allow, or block Xenu Link Sleuth traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.065%
Technical details
- Name
- Xenu Link Sleuth
- Pattern
Xenu Link Sleuth- Tags
- seo
- Reference
- http://home.snafu.de/tilman/xenulink.html
- Added
- 2017/08/19
- Instances
- 1 known sample(s)
Sample User-Agent strings
Xenu Link Sleuth/1.3.8
Block this crawler
robots.txt — disallow Xenu Link Sleuth:
User-agent: Xenu Link Sleuth
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Xenu Link Sleuth [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "Xenu Link Sleuth") {
return 403;
}
← back to all crawlers