The new whitehouse.gov web site embodies the openness and accessibility of Obama’s administration. There’s now an official white house blog, displayed prominently on the home page. The blog doesn’t allow comments, but that’s understandable, since it would be a huge target for comment spammers and would require constant monitoring.
In addition to the blog RSS feed, there are also RSS feeds for agenda articles, press office, photo gallery, and videos.
User-agent: * Disallow: /includes/
The Bush-era robots.txt was over 2300 lines long, blocking search engines from almost all of the site.