C chinomoreno New member Nov 9, 2015 #21 eagleeye dental said: What is robots.txt? Click to expand... Is used for search engines purpose. It is commanding search engine crawlers on what to crawl and what not to crawl.
eagleeye dental said: What is robots.txt? Click to expand... Is used for search engines purpose. It is commanding search engine crawlers on what to crawl and what not to crawl.
