Domains Project
Domains Project web crawler bot
About this crawler
Domains Project is a web crawler identified by the regular-expression pattern Domains Project\/ in the User-Agent request header. It is categorised as seo. Use the regex above to detect, log, allow, or block Domains Project traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.26%
Technical details
- Name
- Domains Project
- Pattern
Domains Project\/- Tags
- seo
- Reference
- https://github.com/tb0hdan/domains
- Added
- 2020/03/16
- Instances
- 1 known sample(s)
Sample User-Agent strings
Mozilla/5.0 (compatible; Domains Project/1.0.3; +https://github.com/tb0hdan/domains)
Block this crawler
robots.txt — disallow Domains Project:
User-agent: Domains Project
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Domains Project\/ [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "Domains Project\\/") {
return 403;
}
← back to all crawlers