heritrix
Internet Archive's Heritrix web crawler framework
About this crawler
heritrix is a web crawler identified by the regular-expression pattern heritrix in the User-Agent request header. It is categorised as archiver. Use the regex above to detect, log, allow, or block heritrix traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.78%
Technical details
- Name
- heritrix
- Pattern
heritrix- Tags
- archiver
- Reference
- https://github.com/internetarchive/heritrix3/wiki
- Added
- 2011/06/21
- rDNS suffixes
.archive.org,.internetarchive.org- Instances
- 22 known sample(s)
rDNS verification (FCrDNS)
Verify a request is genuinely heritrix with forward-confirmed reverse DNS: the client IP's PTR record must end in one of the suffixes below and a forward A/AAAA lookup of that hostname must return the same IP. UA strings alone are spoofable; FCrDNS is not.
.archive.org.internetarchive.org
Sample User-Agent strings
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
+ 2 more samples in crawlers.json
Block this crawler
robots.txt — disallow heritrix:
User-agent: heritrix
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} heritrix [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "heritrix") {
return 403;
}
← back to all crawlers