Sitemap Generator
Check-Domains sitemap generator
About this crawler
Sitemap Generator is a web crawler identified by the regular-expression pattern Sitemap Generator in the User-Agent request header. It is categorised as seo. Use the regex above to detect, log, allow, or block Sitemap Generator traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.065%
Technical details
- Name
- Sitemap Generator
- Pattern
Sitemap Generator- Tags
- seo
- Reference
- http://www.check-domains.com/sitemap/index.php
- Added
- 2026/05/09
- Instances
- 2 known sample(s)
Sample User-Agent strings
Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)
Mozilla/5.0 (compatible; Sitemap Generator/1.3; http://www.check-domains.com/sitemap/index.php) Gecko Check-domains)
Block this crawler
robots.txt — disallow Sitemap Generator:
User-agent: Sitemap Generator
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Sitemap Generator [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "Sitemap Generator") {
return 403;
}
← back to all crawlers