A robots.txt record is a content document in a straightforward arrangement which offers data to web robots, (for example, web index arachnids) about which parts of your site they are and aren't permitted to visit.
On the off chance that you don't have a robots.txt then a web robots will expect...