Speaking the language of robots (How to make robots.txt)

Speaking the language of robots Thursday, March 27, 2008 at 1:27 PM Written by Sean Harding, Webmaster Tools Team We all know how friendly Googlebot is. And like all benevolent robots, he listens to us and respects our wishes about parts of our site that we don't want crawled. We can just give him a robots.txt file explaining what we want, and he'll happily comply. But what if you're intimidated by the idea of communicating directly with Googlebot? After all, not all of us are fluent in the language [ ... ]