What is robots.txt ?

[kalmakuran]

Hello friends,

Please tell me, What is robots.txt ?

 

[salim11]

robots.txt is text file that search information allow or not.

Fastmailsender

 

[Adam Dalena]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

 

[NetgainsTech]

robots.txt is text file that is used to block some of pages which you did not want to crawled by the google bot.

 

[JackA]

The Robots Exclusion Protocol. It is a text file you put on your site to tell search robots which pages you would like them not to visit.

 

[edwina]

Robot.txt:
It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.

 

[LITTLE JEMS]

Robots.txt is text file that is used to allow and disallow.

 

[geeksonrepair]

This file helps to give the information that how pages are indexed and crawl.

 

[arsalan1101]

Robots.txt is common name of a text file that is uploaded to a Web site root directory and linked in the html code of the Web site.
That is used to allow and disallow.

 

[somnathji]

Robot text file is used for dynamic websites and even php websites you can use them
why use them because if you dont want to give any access to google

 

[charvinair]

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site.

 

[fix.97]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

 

[adi123]

This file informs search engine spiders about the data to be crawled.

 

[cacbonro]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

 

[davidweb09]

Robots.txt file is used to control your site which part of your site is indexed by Google and which is not.

 

[Hina Kumari]

If you do not want that a webpage of your site should not be crawled then you could use robot.txt file

 

[mahisharma2]

Robots.txt is text file that is used to allow and disallow.
This is a onpage seo technique.

 

[jaysh4922]

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

 

[northerncarspares]

robot txt is a tools that allow and disallow your site to index

 

[Balajicouriercargo]

This robots.txt file would allow the "googlebot", which is the search engine spider of Google, to retrieve every page from your site except for files from the "cgi-bin" directory. All files in the "cgi-bin" directory will be ignored by googlebot.