what is the difference between meta robot tags and robot. txt?

The main difference in between Meta tag & Robots.txt is Using Meta Robots tag we can block any page from crawling or from indexing and Robots.txt will be useful to perform big task like restricting whole directory from crawling.
 
Robots.txt is a text file that contain instructions for search engine spiders. The text file lists webpages to allow and disallow Google crawling. This helps to control the crawlers on your website.
 
Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, Pandecta Magazine's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider (robot) to ignore it. The Robots META tag does exactly the same thing as the robots.txt file - but it is not as reliable. Not all robots honor the robots meta tag. Use it if your site is in a subdirectory like freewebspace.com/users/mycoolhomepage and you can't get the server administrator to add (or add changes to) a robots.txt file.
 
The Robots Exclusion Protocol (REP) is a group of web standards that regulate web robot behavior and search engine indexing.

REP tags (noindex, nofollow, unavailable_after) steer particular tasks of indexers, and in some cases (nosnippet, noarchive, noodp) even query engines at runtime of a search query. Other than with crawler directives, each search engine interprets REP tags differently.
 
You must log in or register to reply here.
Back
Top