You posted on Scripting News today a link about the MSNBot that's crawling the Internet. Here's the scoop. MSN Search is currently the #3 search engine by queries worldwide. It's also the most profitable unit at Microsoft by headcount (they have less than 50 people and did $150 million in profit last year).
They currently outsource all their searches to Inktomi and LookSmart, with ads embedded through a partnership with Overture. Now that Inktomi is owned by Yahoo (the #2 search engine), they feel it's in their best interests to build their own infrastructure.
They have Google in their crosshairs. They are trying to replicate every feature of Google in the next year, and have tie-ins with the next version of IE (including a 'search' box right on the browser toolbar that by default points at MSN Search) and in Windows Longhorn (the Search function in the shell will have an Internet option that will go to MSN by default). They made a build-vs-buy decision in the last few months (in fact they made an informal offer to purchase Google, which was refused).
The current MSNBot was built by one junior guy at MS as an experiment. So if it does strange things like re-requesting the same data a bunch of times. It's just a research project and not real (yet).
I learned this information on a recent interview loop. I was not asked to sign an NDA, and I got this information from the GM of the MSN Search team. I did not take the job offered by that group.
If you post this information publically or forward it around, please keep my name out of it (i.e. I'm an anonymous source :)