April 17, 2003
Throttle that web server!
Overnight I discovered a certain server down in New Zealand has seen fit to start trying to spider one of my servers. The idiots. They're spidering one of their own sites that has an external link to one of mine. Their spider is making it worse by somehow bastardizing the URLs and recursing into non-existent subdirectories. So I'm seeing all sorts of wasted bandwidth and a generally cluttered up server log (~512k so far).
UPDATE: The admins on the box have contacted me. They've shut down the spider (htdig) and offered an apology. Way to go folks! Running these machines can be a tough job. I hope they have some luck getting it reconfigured properly.
What to do? Well, using apache deny directives is a good start. Trouble is their spider must not be alone. Another one of their hosts, from a different subnet, is likewise making these erroneous requests. So if I block entire subnet ranges or individual IP addresses I'm potentially faced with a lot of work. I'm also faced with other legitimate users on those subnets getting blocked. I'd heard about
mod_throttle some time ago. It's been on my 'to check out' list for ages. Now, it seems, I have need for it.
Initial setup looks good. I've got it blocking IP addressess if they request 'too often'. I'm sure I'll have to tweak these numbers a bit. Not to mention refine which directories get this treatment and which don't. As I get a better grip on it I'll be sure to report my experiences.
