Best Robots.txt

[eyecold]

Robots.txt is very important to control the indexing and crawling of search engines and other bots. Tell me what you think should be best practices for a robots.txt file. Should I allow every body to crawl the website or should I only allow certain hosts or search engines to visit my website. Please share with me a sample of best robots.txt file of you have.

 

[Prince25]

Robots.txt files are good to have in Website root source. These files are good to give direction to search engine crawlers about which page of site is to crawl or index and which need not.

 

[easyarticles]

You allow search to crawl or index the pages which is seen by user, if your pages no content or images and not much helpful for user than you disallow that page in robots.txt. For example some website will popup the signup page, we disallow that page, it only for login to your website.

 

[OhioTom76]

You should at least have a Robots.txt file, even if there is nothing in it currently. Otherwise you may get error messages from the engines in their Webmaster tools. You will need to use it if you have private directories that you don't want indexed publicly, such as if you are developing a new section of your site that isn't ready to go live yet. You would also need to use the file if you move a lot of pages on your site and need to redirect the spiders to the correct pages, so you don't lose rankings.

 

[deven75]

By creating a robots.Txt file you can easily control the crawling access as per you needs some time websites handling person create multiple pages or folders to which are not required later or need for general guidance or can even have private massage, As we know that almost all the all the search engines follows the instruction in Robots.
Example for Robots.Txt
-------
User-agent: *

Disallow: /groups
Disallow: /images
Disallow: /catalogs
Disallow: /catalogues

-------

Before creating a file please optimize the content and folders because wrong insertion of folder or pages can damage you SEO efforts.

 

[seowork]

hi
thanks eyecold for this information sharing ..........

 

[eximpulse]

Hi deven,

Thanks for sharing that help full information.

 

[marvinsam]

Robot.txt are HTML files good to have in Website root source. These files are good to give direction to search engine crawlers about which page of site is to crawl or index and which need not.

 

[salim11]

I agree with this post.

 

[shashant]

Robot.txt are HTML files that inform the crawler that this page of site is not to crawl or index

 

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

 

[sharmaroshni]

The file lists webpages that are allowed and disallowed from search engine crawling.