What is the use of Robots.txt?

[smitharetha]

What is the use of Robots.txt?

 

[Fadia Sheetal]

Robots.txt is for to stop the search engine spider to crawl the website or web page.

 

[smitharetha]

Robots.txt is for to stop the search engine spider to crawl the website or web page.

thank you ^^

 

[auraiyaitsolutions]

Robots.txt is a set of instructions for search engines to index your site.

 

[tpmike]

Robots.txt file tells the search engines spider which pages to access and index on your website and which pages not...

 

[Adam Dalena]

Robots.txt is for to stop the search engine spider to crawl the website or web page.

 

[RH-Calvin]

Robots.txt is a text file defined in the website to contain instructions for search engine spiders. They list webpages that are allowed and disallowed from search engine crawling.

 

[mehreenshaikh]

Hii

-Robots.txt is best on page activity.
-Are used our site any page block.Block for any no-follow page are used.

 

[arindamdutta16]

The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

 

[Pillars]

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

 

[tbionline]

It stops search engine to crawl unwanted or confidential pages.

 

[orientation]

robots.txt is a text file which gives instructions to google which page to crawl