Linkwalker and BDFetch

Noticing that Linkwalker requests robots.txt and is followed by BDFetch.

Example of the Linkwalker robot:

72.14.164.165 - - "GET /robots.txt HTTP/1.1" 406 1326 "www.seventwentyfour.com/" "LinkWalker/2.0"

Examples of the BDFetch robot:

72.14.164.177 - - "GET /robots.txt HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.193 - - "GET /index.html HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.173 - - "GET /robots.txt HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.191 - - "GET /index.html HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.177 - - "GET /robots.txt HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.188 - - "GET /index.html HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"

Solution to bandwidth gobbling bots in this IP range:

Add these to the deny list on the router.
72.14.163.0/24
72.14.164.0/24

Written by admin on September 28th, 2009 with comments disabled.
Read more articles on How To and Spiders.

• [+] Digg: Feature this article
• [+] Del.icio.us: Bookmark this article
• [+] Furl: Bookmark this article

Related articles

• How To Block Proxy Visitors From Your Web Site (November 3rd, 2009)
• HOW TO: Fix Lock table is out of available locker entries (January 29th, 2009)
• HOW TO Check Port Speed On Linux Server (August 22nd, 2008)
• HOW TO Check Your Server’s RAM Status. (August 10th, 2008)
• How To Ban “User-Agent:” Scrappers (July 4th, 2008)