Robots.txt Generator - Through the robots.txt file, search engines receive signals about what can be accessed and what is not within a website. The robots.txt file is uploaded to the root of the site. It is a simple text file that uses a page selection parameter to be analyzed or not by the search engine robots (User-agent & Disallow).