AliceFowell89
New member
Hi,
My question is, Is it necessary to use Robots.txt?
My question is, Is it necessary to use Robots.txt?
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Is it necessary to use Robots.txt?
- Thread starter AliceFowell89
- Start date
iambhevendra
New member
The robots.txt protocol is a useful tool to prevent certain bots from spidering certain pages/sections of sites while allowing others through. It's also a lot easier to modify a robots.txt file than it is to update dozens, hundreds or even thousands of Web pages to make a single change. Also, unlike the META tag, the robots.txt file can block SPECIFIC search engine spiders from crawling and indexing your site.
And has already been mentioned, not having a robots.txt file can clutter your server logs with needless 404 error returns.
And has already been mentioned, not having a robots.txt file can clutter your server logs with needless 404 error returns.
Alanah_Rae
New member
The robots.txt protocol is a useful tool to prevent certain bots from spidering certain pages/sections of sites while allowing others through. It's also a lot easier to modify a robots.txt file than it is to update dozens, hundreds or even thousands of Web pages to make a single change. Also, unlike the META tag, the robots.txt file can block SPECIFIC search engine spiders from crawling and indexing your site.
And has already been mentioned, not having a robots.txt file can clutter your server logs with needless 404 error returns.
great Information. Thanks for sharing !!
LisaHandson89
New member
It is not important use robots. txt file but it is advisable to use it, if there is any page on your websites which you don't want search engine to crawl. Robots.txt also improves better indexing for the website.
The robots.txt protocol is a useful tool to prevent certain bots from spidering certain pages/sections of sites while allowing others through. It's also a lot easier to modify a robots.txt file than it is to update dozens, hundreds or even thousands of Web pages to make a single change. Also, unlike the META tag, the robots.txt file can block SPECIFIC search engine spiders from crawling and indexing your site.
And has already been mentioned, not having a robots.txt file can clutter your server logs with needless 404 error returns.
And has already been mentioned, not having a robots.txt file can clutter your server logs with needless 404 error returns.