Where do you put your robots.txt?

The robots.txt file is placed in your www or public_html directory.

So the correct location where to put this file is in yourdomainname.com/robots.txt. This file indicates how "deep" the spiders can crawl into your website.
 
- its used to Block websites part/ Pages for search engine if you add any pages or directory of websites in robots.txt any crawler will not crawler that page or part of sites.
- It works likes this: a robot wants to vists a Web site URL, say www.abc.com/robots.txt
- robots.txt code -

User-agent: *
Disallow: /

- save this code file site root.
 
Robots.txt is a text file that is used to define in your website. It contains instructions for search engine spiders to crawl and disallow certain webpages in search engines. They help to control the crawling of your website.
 
Yes, save it under the root folder, or if you are feeling to change different roles for pages like you want some to index and follow but not for others so you can use meta tags for each pages instead of robot.txt file

<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
 
njproperty said:
Hi friends!

Where do you put your robots.txt?
Click to expand...

Robots.txt is placed in the root of the website. It is first file robots look after to know which pages are allowed to crawl and number of pages which are blocked from crawling. If you don't want to add Robots.txt file you can add Robots tag as well. The only difference between Robots.txt file and robots tag is tag is required to be added in each and every page of the website, whereas Robots.txt file is added once only.
 
You must log in or register to reply here.
Back
Top