parse.ly scraper
Content analytics platform crawler for metadata extraction
About this crawler
parse.ly scraper is a web crawler identified by the regular-expression pattern parse\.ly scraper\/ in the User-Agent request header. It is categorised as seo. Use the regex above to detect, log, allow, or block parse.ly scraper traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.065%
Technical details
- Name
- parse.ly scraper
- Pattern
parse\.ly scraper\/- Tags
- seo
- Reference
- https://www.parse.ly/
- Added
- 2026/04/17
- Instances
- 1 known sample(s)
Sample User-Agent strings
Mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com)
Block this crawler
robots.txt — disallow parse.ly scraper:
User-agent: parse.ly scraper
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} parse\.ly scraper\/ [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "parse\\.ly scraper\\/") {
return 403;
}
← back to all crawlers