what is the robot.txt

Robot.txt:
It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.
 
Robots.txt file - is an extraordinary administration document, which the vast majority of the internet searchers robots demand while going to a webpage. With help of this record you can close from indexing both separate pages and the entire segments of a site. In this document likewise can be set connections to XML sitemaps for web crawler robots. It helps robots to file site right.
 
Hello Dear,

- The robots exclusion standard, otherwise called the robots exclusion protocol or simply robots.txt, is a standard utilized by sites to communicate with web crawlers and other web robots.
- The standard determines how to inform the web robot about which zones of the site should not be processed or filtered.
- It's Useful Activity for Promotion.
- It's Part of the SEO On-Page Optimization Activity.
- It's Good for Increase traffic.


Thanks.
 
Robot.txt is a text file, you can write in notepad and word pad with robot.txt extension. If you don't want to crawl your file or folder you just put the Robot in it. So search engine doesn't going to crawl that file or folder.
 
A robots.txt file,is a protocol to search engines, it indicates to web crawlers that they are to avoid going into URLs beginning with /search on that host. Your robots.txt file is related to the Robots Exclusion Standard.
 
niu_noida said:
what is the robots.txt. and what is the advantage of robots.txt
Click to expand...

Robots.txt file is placed in root directory, which helps to tell search engine list of pages that are allowed to crawled and pages which are blocked from crawling. It is suggested to have XML file at the end of each robots.txt file.

If you don't want to add robots.txt file than you can add robots tag in the each page.
 
You must log in or register to reply here.
Back
Top