ApifyWebsiteContentCrawler

ai-crawler ApifyWebsiteContentCrawler

Apify's full website content extractor

About this crawler

ApifyWebsiteContentCrawler is a web crawler identified by the regular-expression pattern ApifyWebsiteContentCrawler in the User-Agent request header. It is categorised as ai-crawler. Use the regex above to detect, log, allow, or block ApifyWebsiteContentCrawler traffic in your web server, CDN edge rules, or robots.txt.

Block-rate · top 25k sites

0.26%

latest snapshot
2026-06-04
matched key: ApifyWebsiteContentCrawler

Technical details

Name: ApifyWebsiteContentCrawler
Pattern: ApifyWebsiteContentCrawler
Tags: ai-crawler
Reference: https://apify.com/apify/website-content-crawler
Added: 2026/04/26
rDNS suffixes: .apify.com
Instances: 1 known sample(s)

rDNS verification (FCrDNS)

Verify a request is genuinely ApifyWebsiteContentCrawler with forward-confirmed reverse DNS: the client IP's PTR record must end in one of the suffixes below and a forward A/AAAA lookup of that hostname must return the same IP. UA strings alone are spoofable; FCrDNS is not.

.apify.com

Sample User-Agent strings

ApifyWebsiteContentCrawler/1.0 (+https://apify.com/apify/website-content-crawler)

Block this crawler

robots.txt — disallow ApifyWebsiteContentCrawler:

User-agent: ApifyWebsiteContentCrawler Disallow: /

Apache .htaccess — return 403:

RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ApifyWebsiteContentCrawler [NC] RewriteRule .* - [F,L]

Nginx — return 403 inside a server block:

if ($http_user_agent ~* "ApifyWebsiteContentCrawler") { return 403; }

← back to all crawlers