walterwaller
New member
why is use robots txt file and how to implement for this ?
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
What is the robots txt file?
- Thread starter walterwaller
- Start date
amarjyotiadvertising
New member
A robot.txt file is a part of your page source that directs the web crawler on how, when and where to visit or crawl on your site. A robot.txt file is seen as an instruction to the web crawler and is thus an important part of your website.
GRONEX PRIYA
New member
The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl.
TangelThor
New member
Hi....!
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned
Thanks.......
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned
Thanks.......
gtechwebindia
New member
Robots.txt is a text file file in which we use code to give instructions to spider that the page is a crawl or not. we use allow or disallow to give instructions.