netEstate NE Crawler
NetEstate web crawler for domain analysis
About this crawler
netEstate NE Crawler is a web crawler identified by the regular-expression pattern netEstate NE Crawler in the User-Agent request header. It is categorised as seo. Use the regex above to detect, log, allow, or block netEstate NE Crawler traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.78%
Technical details
- Name
- netEstate NE Crawler
- Pattern
netEstate NE Crawler- Tags
- seo
- Reference
- http://www.website-datenbank.de/
- Added
- 2016/01/23
- Instances
- 2 known sample(s)
Sample User-Agent strings
netEstate NE Crawler (+http://www.sengine.info/)
netEstate NE Crawler (+http://www.website-datenbank.de/)
Block this crawler
robots.txt — disallow netEstate NE Crawler:
User-agent: netEstate NE Crawler
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} netEstate NE Crawler [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "netEstate NE Crawler") {
return 403;
}
← back to all crawlers