What is the function of the Robots.txt file?

[nancyisabell]

What is the function of the Robots.txt file?

 

[WebOutGateway]

Hi! Robot.txt is a text file which has the syntax understandable by search engines. It allows search engine spiders to crawl your website.

 

[edwina]

Robot.txt file is a text file placed at the root of a website .
It contains instructions for bots crawling a website.

 

[jecika]

A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers.

 

[Layla Smart]

Robot text is useful is an effective way to communicate with Search Engines (bolts or crawlers). If you don't want a Search Engine to crawl a specific (set) of Page(s) you can list those link on Robot text and tell SEs not to crawl.

You can also use Robot Text to prevent Search Engines from crawling your entire website.

 

[RH-Calvin]

Robots.txt is a text file that contains instructions for search engine crawlers, instructing to crawl and not to crawl certain webpages of a website.

 

[vicky_patel]

Robots.txt is using to stop the search engines crawling our web-site. It is effective file for the site and it is not a plugin. It is coding file.

 

[randiv]

i think robots is the instructor of website for search engine crawler.

 

[Kevinslagus]

robots.txt file gives direction for search engine crawler about particular page or directory whether he has a permission to crawl or not.

 

[robertden]

Robots.txt file is used to give instruction to crawlers/searchbots. For example which website page/ directories we want to crawl, follow, index etc.

it's format is

User-agent: * (* means all search bots like Googlebot, scooter etc.)
Disallow: / (By Disallowing we are restricting searchbots not to follow any or all page or folder. For eg: wp-admin, wp-include etc.)
Allow: ( allowing means allow all searchbots to crawl and index all web pages)

 

[smith1wholesale]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit..

 

[lomavic]

the file that's make google don't see some pages on your website and see others

 

[auraiyaitsolutions]

to give instructions to Google spiders that which part of website should be cached & which is not.

 

[esourceparts]

Robots.txt is a text file which is to be uploaded into the root directory of the website which decides whether to allow or disallow the spiders/crawlers to crawl the website or the particular page.

 

[WebOutGateway]

Hi! Robot.txt is a text file which has the syntax understandable by search engines. It gives a command to the crawler on which page is to avoid and which page is to index.

 

[jyoti457]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

 

[Balwinder Singh]

It is useful for search engines to crawl a website page or not.

 

[jasmynroi]

Robot text file is an essential part of site from crawlers point of view you may save all those files in it taht you dont want to read by crawlers.

Send Rakhi to India | Send Rakhi Gifts to India | Buy Rakhi Online

 

[brainmineae]

Robot.txt is a text file by means of which search engine understand which part of your website should be crawled and which should not be.

 

[dombowkett]

Robots.txt file control website indexing from Google & other search engines.