CMS Crawler
CMS detection and analysis
About this crawler
CMS Crawler is a web crawler identified by the regular-expression pattern CMS Crawler in the User-Agent request header. It is categorised as scanner. Use the regex above to detect, log, allow, or block CMS Crawler traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.065%
Technical details
- Name
- CMS Crawler
- Pattern
CMS Crawler- Tags
- scanner
- Reference
- http://www.cmscrawler.com
- Added
- 2026/05/03
- Instances
- 1 known sample(s)
Sample User-Agent strings
Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
Block this crawler
robots.txt — disallow CMS Crawler:
User-agent: CMS Crawler
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} CMS Crawler [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "CMS Crawler") {
return 403;
}
← back to all crawlers