CC
content crawler spider
Content Crawler web crawler for discovery
About this crawler
content crawler spider is a web crawler identified by the regular-expression pattern content crawler spider in the User-Agent request header. It is categorised as seo. Use the regex above to detect, log, allow, or block content crawler spider traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.065%
Technical details
- Name
- content crawler spider
- Pattern
content crawler spider- Tags
- seo
- Added
- 2013/01/11
- Instances
- 0 known sample(s)
Sample User-Agent strings
no public sample user-agents recorded.
Block this crawler
robots.txt — disallow content crawler spider:
User-agent: content crawler spider
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} content crawler spider [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "content crawler spider") {
return 403;
}
← back to all crawlers