The National Archives UK
UK National Archives web archive
About this crawler
The National Archives UK is a web crawler identified by the regular-expression pattern The National Archives UK in the User-Agent request header. It is categorised as archiver. Use the regex above to detect, log, allow, or block The National Archives UK traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.065%
Technical details
- Name
- The National Archives UK
- Pattern
The National Archives UK- Tags
- archiver
- Reference
- http://www.nationalarchives.gov.uk/webarchive/
- Added
- 2026/05/09
- Instances
- 1 known sample(s)
Sample User-Agent strings
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36 The National Archives UK Government Web Archive: http://www.nationalarchives.gov.uk/webarchive/; [email protected]
Block this crawler
robots.txt — disallow The National Archives UK:
User-agent: The National Archives UK
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} The National Archives UK [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "The National Archives UK") {
return 403;
}
← back to all crawlers