What is the use of Robots.txt?

[CarolineMurphy89]

What is the use of Robots.txt?

 

[sammystorm]

Web site owners use the /robots.txt file to give instructions about their site to web robots

 

[sonikathakur]

A robots.txt file is placed at the root of your domain to tell search engines and other web robots or spiders how to interact with your site. The robots.txt file allows you to specify which sections or pages of your site you want search engines to access, and which sections you want to restrict access to.

 

[edwina]

Robot.txt:
It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.

 

[Broadanderson]

A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file. It helps to avoid hacking problems to a website for payment gateway sites.

 

[towertech]

Hi,

Robots.txt file indicate to search engine crawler to which part of website you want to allow to crawl and which part of your website you don;t want to crawl.

Thanks

 

[jandesigner1]

Robots.txt helps to prevent pages which you don't want to crawl such as admin pages..