This post over on IBM's phpblog@developerWorks from Ken Coar talks about PHP and robots.txt.

The week before last I was in Montreal, Quebec, Canada for PHP Quebec Conference 2005. One of the presentations I gave was about using PHP scripts instead of normally static files like robots.txt, and it gratifyingly raised a couple of eyebrows.

Making robots.txt a dynamic file can have a definite impact on performance, but it lets you answer queries a little more specifically. [...] So if robots.txt is a dynamic file, it can handle Type III malbots by only responding with a single client's permissions — those of the client to whose request it's responding.

His suggestion to make the robots.txt file dynamic is one of those things that most people just don't think about - but when they hear about it, they're stopped and wonder why they didn't think of it sooner. It can be used to make "spider traps" for the incoming clients, keeping them out of various protected areas...