archive.org_bot

archiver archive\.org_bot

Archive.org web crawler for preservation

About this crawler

archive.org_bot is a web crawler identified by the regular-expression pattern archive\.org_bot in the User-Agent request header. It is categorised as archiver. Use the regex above to detect, log, allow, or block archive.org_bot traffic in your web server, CDN edge rules, or robots.txt.

Block-rate · top 25k sites

1.7%
latest snapshot
2026-06-04
matched key: archive.org_bot
2026-05-012026-06-042.3%

Technical details

Name
archive.org_bot
Pattern
archive\.org_bot
Tags
archiver
Reference
http://archive.org/details/archive.org_bot
Added
2015/04/14
rDNS suffixes
.archive.org, .internetarchive.org
Instances
6 known sample(s)

rDNS verification (FCrDNS)

Verify a request is genuinely archive.org_bot with forward-confirmed reverse DNS: the client IP's PTR record must end in one of the suffixes below and a forward A/AAAA lookup of that hostname must return the same IP. UA strings alone are spoofable; FCrDNS is not.

Sample User-Agent strings

Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)

Block this crawler

robots.txt — disallow archive.org_bot:

User-agent: archive.org_bot Disallow: /

Apache .htaccess — return 403:

RewriteEngine On RewriteCond %{HTTP_USER_AGENT} archive\.org_bot [NC] RewriteRule .* - [F,L]

Nginx — return 403 inside a server block:

if ($http_user_agent ~* "archive\\.org_bot") { return 403; }
← back to all crawlers