How To Ban “User-Agent:” Scrappers

Lately, there has been a huge amount of annoying website scrappers combing through our sites using the useragent below.

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

Obviously, legitimate human visitors do not have the term “User-Agent:” in their useragent field. Months ago I had added a ModSecurity rule to help identify and block these bandwidth wasters and copyright infringer’s.

Example of log details:

69.14.204.163 – - [04/Jul/2008:13:09:58 -0400] “GET / HTTP/1.1″ 410 317 “-” “User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”

Thumbs Down To "User-Agent" Scrappers

STEPS

1) Using ModSecurity 2.0+, I instituted this rule to modsec2.conf located in usr/local/apache/conf/

SecRule HTTP_User-Agent “User-Agent” “deny,log,status:410″

2) Restart Apache

/sbin/service httpd restart

Written by admin on July 4th, 2008 with comments disabled.
Read more articles on How To and ModSecurity.