A robots.txt file in your web site’s root directory informs robots who follow the Robots Exclusion Protocol on which areas of your site can be indexed and which areas cannot be indexed.
A simple robots.txt implementation tells all robots that your entire site is indexable:
User-agent: *
Disallow:
But be careful not to do the following. The slash “/” tells the robots that the entire site is off limits:
User-agent: *
Disallow: /
