Screaming Frog SEO Spider
Screaming Frog SEO tool web crawler
About this crawler
Screaming Frog SEO Spider is a web crawler identified by the regular-expression pattern Screaming Frog SEO Spider in the User-Agent request header. It is categorised as seo. Use the regex above to detect, log, allow, or block Screaming Frog SEO Spider traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.46%
Technical details
- Name
- Screaming Frog SEO Spider
- Pattern
Screaming Frog SEO Spider- Tags
- seo
- Reference
- https://www.screamingfrog.co.uk/seo-spider/
- Added
- 2016/01/08
- rDNS suffixes
.screamingfrog.co.uk- Instances
- 1 known sample(s)
rDNS verification (FCrDNS)
Verify a request is genuinely Screaming Frog SEO Spider with forward-confirmed reverse DNS: the client IP's PTR record must end in one of the suffixes below and a forward A/AAAA lookup of that hostname must return the same IP. UA strings alone are spoofable; FCrDNS is not.
.screamingfrog.co.uk
Sample User-Agent strings
Screaming Frog SEO Spider/5.1
Block this crawler
robots.txt — disallow Screaming Frog SEO Spider:
User-agent: Screaming Frog SEO Spider
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Screaming Frog SEO Spider [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "Screaming Frog SEO Spider") {
return 403;
}
← back to all crawlers