robots.txt..?

[sofiajain]

Hello Friends,

Please tell me, What is robots.txt..? How are Benefits..?

 

[RH-Calvin]

Robots.txt is a text file in your website that defines webpages instructing search engine spiders to crawl and disallow them. It is helpful to control the crawling activity of your website.

 

[hazelkeech]

robots.text is webmaster's created text file, which instruct the robots how to index the page and crawl on your website.

 

[jamesnader]

robots.txt file give instruction about their website to web indexing. It's called Robots Exclusion Protocol. Robots.txt file you put on your website to tell crawler which page would you like to crawl.

 

[CarolinaFlores]

There is more reason to using a robots.txt file other than telling bots which pages and directories not to crawl.