Is Robots allow or not to all Search Engines ?

[Fadia Sheetal]

Hay Users,

I start working on a Website and bit confuse about the robots. What is the right extension for search engines allow or not to crawl the website ?

 

[Sophia]

When Disallow with / than that is calling allow to search engine spiders & when allow with / means crawling permission for search engines.

 

[RH-Calvin]

Yes robots.txt file contains instructions for all search engines. It is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.