How strong is your server kung fu?

September 28th, 2009

You are currently browsing the articles from Server Fu written on September 28th, 2009.

Linkwalker and BDFetch

Noticing that Linkwalker requests robots.txt and is followed by BDFetch.

Example of the Linkwalker robot:

72.14.164.165 - - "GET /robots.txt HTTP/1.1" 406 1326 "www.seventwentyfour.com/" "LinkWalker/2.0"

Examples of the BDFetch robot:

72.14.164.177 - - "GET /robots.txt HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.193 - - "GET /index.html HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.173 - - "GET /robots.txt HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.191 - - "GET /index.html HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.177 - - "GET /robots.txt HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"
72.14.164.188 - - "GET /index.html HTTP/1.1" 5816 "www.brandimensions.com" "BDFetch"

Solution to bandwidth gobbling bots in this IP range:

Add these to the deny list on the router.
72.14.163.0/24
72.14.164.0/24

Written by admin on September 28th, 2009 with comments disabled.
Read more articles on How To and Spiders.