Dow Jones Searchbot
Dow Jones news and financial content aggregation crawler
About this crawler
Dow Jones Searchbot is a web crawler identified by the regular-expression pattern Dow Jones Searchbot in the User-Agent request header. It is categorised as monitoring. Use the regex above to detect, log, allow, or block Dow Jones Searchbot traffic in your web server, CDN edge rules, or robots.txt.
Block-rate · top 25k sites
0.056%
Technical details
- Name
- Dow Jones Searchbot
- Pattern
Dow Jones Searchbot- Tags
- monitoring
- Reference
- https://www.dowjones.com/
- Added
- 2026/04/26
- Instances
- 1 known sample(s)
Sample User-Agent strings
Mozilla/5.0 (compatible; Dow Jones Searchbot)
Block this crawler
robots.txt — disallow Dow Jones Searchbot:
User-agent: Dow Jones Searchbot
Disallow: /
Apache .htaccess — return 403:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Dow Jones Searchbot [NC]
RewriteRule .* - [F,L]
Nginx — return 403 inside a server block:
if ($http_user_agent ~* "Dow Jones Searchbot") {
return 403;
}
← back to all crawlers