VV's Corner » SEO http://vinoaj.com/blog my collection of random thoughts, how-to's and code snippets Sat, 04 Dec 2010 15:05:54 +0000 en hourly 1 http://wordpress.org/?v=3.1 robots.txt http://vinoaj.com/blog/2007/04/robotstxt/ http://vinoaj.com/blog/2007/04/robotstxt/#comments Fri, 06 Apr 2007 08:02:38 +0000 admin http://vinoaj.com/blog/howtos/seo/robots/robotstxt/ A robots.txt file in your web site’s root directory informs robots who follow the Robots Exclusion Protocol on which areas of your site can be indexed and which areas cannot be indexed.

A simple robots.txt implementation tells all robots that your entire site is indexable:
User-agent: *
Disallow:

But be careful not to do the following. The slash “/” tells the robots that the entire site is off limits:
User-agent: *
Disallow: /

]]> http://vinoaj.com/blog/2007/04/robotstxt/feed/ 0