] Many so called webbots or web spiders are currently used
] for many different things on the Internet. Examples
] include search engines that use them to catalog the
] Internet, email marketing people that search for email
] addresses and many more.
]
] Some of those robots are welcome, others are not. This
] page will show you how I catch the bad ones, and how I
] stop them from bothering me again.

A few neat techniques for detecting and trapping "bad" webrobots. Most are reactionary, and quite frankly stupid.

First of all, if, say tomorrow, Google's crawler googlebot suddenly decided to ignore robots.txt, what would you do? Ban googlebot? HA! No, of course not. Big companies aren't gonna ban any robot that ignores robots.txt. It costs them no more in bandwidth, and maybe its a new robot that they *do* want indexing stuff. Small websites can ban me if they want, if the admin's have time to sit around reading logs and banning IPs, then they aren't spending their time doing anything cool, so who cares if they ban me.

How to keep bad robots