BrightEdge Crawler
BrightEdge web crawler for SEO
About this crawler
BrightEdge Crawler is a web crawler identified by the regular-expression pattern BrightEdge Crawler in the User-Agent request header. It is categorised as seo. Use the regex above to detect, log, allow, or block BrightEdge Crawler traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.39%
Technical details
- Name
- BrightEdge Crawler
- Pattern
BrightEdge Crawler- Tags
- seo
- Reference
- https://www.brightedge.com/
- Added
- 2023/09/08
- rDNS suffixes
.brightedge.com- Instances
- 1 known sample(s)
rDNS verification (FCrDNS)
Verify a request is genuinely BrightEdge Crawler with forward-confirmed reverse DNS: the client IP's PTR record must end in one of the suffixes below and a forward A/AAAA lookup of that hostname must return the same IP. UA strings alone are spoofable; FCrDNS is not.
.brightedge.com
Sample User-Agent strings
BrightEdge Crawler/1.0 ([email protected])
Block this crawler
robots.txt — disallow BrightEdge Crawler:
User-agent: BrightEdge Crawler
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} BrightEdge Crawler [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "BrightEdge Crawler") {
return 403;
}
← back to all crawlers