Lijit Crawler
Lijit web crawler
About this crawler
Lijit Crawler is a web crawler identified by the regular-expression pattern Lijit Crawler in the User-Agent request header. It is categorised as seo. Use the regex above to detect, log, allow, or block Lijit Crawler traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
No block-rate data for this crawler.
Technical details
- Name
- Lijit Crawler
- Pattern
Lijit Crawler- Tags
- seo
- Reference
- http://www.lijit.com/robot/crawler
- Added
- 2026/05/09
- Instances
- 1 known sample(s)
Sample User-Agent strings
Lijit Crawler (+http://www.lijit.com/robot/crawler)
Block this crawler
robots.txt — disallow Lijit Crawler:
User-agent: Lijit Crawler
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Lijit Crawler [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "Lijit Crawler") {
return 403;
}
← back to all crawlers