What is robots.txt? Is used for search engines purpose. It is commanding search engine crawlers on what to crawl and what not to crawl.