MSNBot, which powers Microsoft's Bing search engine, seems to have a different understanding of what is reasonable spider behavior.
It is quite common for MSNBot to request "/robots.txt" 5 or 6 times within a few seconds on any given Web site. I have seen it request URLs that have never existed and seem to be totally made up or applicable to some other site. And, in general, it tends to request pages too frequently and with too many repeat requests in a short amount of time. I have also seen MSNBot request pages or areas of the site that are specifically disallowed in robots.txt files.
I have written to MS on this and still have got no definite reply as to how to control msn/bing bot. At least google put out controls that you can use to control their bot. Good luck Dallan, if I stumble across anything on our sites for work I'll let you know.