what is robots.txt file?

robots.txt file:

Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling.
 
Back
Top