Recovery.gov Search Engine Conspiracy?

CNET reports that the new government website Recovery.gov has been using a special file to prevent being indexed by search engines. The robots.txt file, which has since been removed, contained the following code to prevent Google, Yahoo, and other reputable search engines from indexing any of it’s content: # Deny all search bots, web spiders […]