The robots.txt is a very simple text file that is placed on your root directory. An example would be www.yourdomain.com/robots.txt. This file tells search engine and other robots which areas of your site they are allowed to visit and index.

At it's most simple, a robots.txt file looks like this:

User-agent: *
Disallow:

This one tells all robots (user agents) to go anywhere they want (disallow nothing).