What does mean by Robot.txt, is it necessary in term of SEO
- Thread starter Anju jha
- Start date
ravikumar007
New member
Robots.txt is a text file you include on your site to inform search robots that which pages should be crawled or not.
Your Robots.txt file is what tells the search engines which pages should be accessed and indexed on your website. For example, if you specify in your Robots.txt file that you don’t want the search engines to crawl a page, that page will not be shown in search engine result page and web users won’t be able to find it. Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.
Your Robots.txt file instructs these programs not to search pages on your site which you mention using a “disallow” command. For example, the following Robots.txt command:
User-agent: *
Disallow: /quiztest
Disallow: /images
Disallow: /asp/demo_db_edit.asp
Disallow: *.htm
Disallow: *.html
Disallow: *.aspx$
Disallow: *.php$
…would prevent all search engine crawlers from crawling the following page on your website:
Your Robots.txt file is what tells the search engines which pages should be accessed and indexed on your website. For example, if you specify in your Robots.txt file that you don’t want the search engines to crawl a page, that page will not be shown in search engine result page and web users won’t be able to find it. Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.
Your Robots.txt file instructs these programs not to search pages on your site which you mention using a “disallow” command. For example, the following Robots.txt command:
User-agent: *
Disallow: /quiztest
Disallow: /images
Disallow: /asp/demo_db_edit.asp
Disallow: *.htm
Disallow: *.html
Disallow: *.aspx$
Disallow: *.php$
…would prevent all search engine crawlers from crawling the following page on your website:
Last edited:
nareshverma
New member
Yes robots.txt text file that is is search engine information allow or not .
Bydefault your site is open for all search engine crawlers. If you want to hide or no index some of your pages of website you can create robots.txt file for that. It helps the web crawlers what to index and what to not. Refer this page for more details: https://moz.com/learn/seo/robotstxt
Last edited:
sadianisar
New member
Yes. It's very important in SEO. Robots.txt is a notepad file in which we write commands to stop the crawler to enter/visit that page. There are only 2 codes which should be mentioned in the file which are
User-agent: * (for which search engine)
Disallow:/aboutus.html (which page you don't want to index/crawl)
User-agent: * (for which search engine)
Disallow:/aboutus.html (which page you don't want to index/crawl)
andrewbenn
New member
a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages