What is the use of Robots.txt?

[rohinisen]

The use of robot txt is a text file that details website and guide the crawler which mention in the search engine.

 

[niu_noida]

Robots...txt gives instruction to crawler . which webpage crawl allowed or Disallowed

 

[Gndronacharya]

Hello,
Robot.txt file help to protect the site. It is a type of coding which use into on page.

 

[arnoldkelly1234]

The /robots.txt checker can check your site's /robots.txt file and meta tags.

 

[ShreyaKoushik23]

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

 

[rohinisen]

Robot txt is the text file that details website that are allowed to the search engine to search for the results.