What is robots.txt?

Robots.txt is a text file which You put it into your site that tell search robots that which pages of your site you would like them not to visit. This files inform search engine spiders that how to interact with indexing your contents.
 
Robot.txt is a text file which we put on the site to tell the search robots that which pages you don't want them to visit.
 
Robots.txt is a file which contains the information for the search engine bot or spider about your site to which page you want to get index and which are not. You must use the robots.txt in your site to get more benefits in SEO. Here is http://www.drinstech.com/robots.txt an example of a robots.txt file.
 
File used to direct or to tell web bots what pages and directories to index or not index. This file must be placed in the root directory on the server hosting your pages. The file should be named robots.txt and should have read permissions. Below is an example of a robots.txt file.
#robots.txt file for www.Example.com
User-agent: *
Disallow: /directory1/
Disallow: /directory2/
 
robots.txt is a file which provide the instruction about webpage to be crawled to googlebot. Googlebot follow the instruction while crawling your website.
So must use robots.txt file for your website.

Thank you.
 
Robots.txt is common name of a text file that is uploaded to a Web site's root directory andlinked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.
 
eweb1 said:
File used to direct or to tell web bots what pages and directories to index or not index. This file must be placed in the root directory on the server hosting your pages. The file should be named robots.txt and should have read permissions. Below is an example of a robots.txt file.
#robots.txt file for Example Domain
User-agent: *
Disallow: /directory1/
Disallow: /directory2/
Click to expand...

Nice post i agree with your post.
Thanks for sharing....
 
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.
 
Robot text is a some program for search engine robot whether visit ,see or crawl the page or not & also have to follow instructions like indexing.
User agent:*
Disallow:
(all allow)
User agent:*
Disallow:/
(not allow for specific path)
visit & read articles related seo in shazzadhossain.blogpost.com
 
Last edited:
All have with nice explain of the Robots.txt file. A robots.txt file is a file at the root of your site that indicates those parts of your site you don't want accessed by search engine crawlers.
 
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
 
Last edited:
You must log in or register to reply here.
Back
Top