robots.txt Disallow

W

web_master_foro

New member
Please guide me how to disallow search engines and spiders and other bad bots to visit my website. These spiders are eating up my website bandwidth. I want to allow only Yahoo, Google and few other search engines to visit my website and crawl and want to block all of the rest.

Please guide me how can i allow few search engines and completely stop and disallow all unwanted crawlers to visit this website.
 
Only good search engines follow whats mentioned in robots.txt file. The bad robots you are talking about will not follow what you want. There is not much you can do about this problem so the only solution you are left is to purchase a good server with a good bandwidth.
 
Every day hundreds of them go out and scour the web, whether it's Google trying to index the entire web, or a spam bot collecting any email address it could find for less than honorable intentions. As site owners, what little control we have over what robots are allowed to do when they visit our sites exist in a magical little file called "robots.txt."
 
I don't think it is possible
Bạn có thể xem qua gói thiết kế website du lịch của chúng tôi để biết được công nghệ thiết kế web mobile chuẩn responsive hàng đầu VN, dịch vụ thiết kế website của chúng tôi đảm bảo chuyên nghiệp mà giá lại rẻ: thiết kế website tại hà nội uy tín, thiết kế website theo yêu cầu, thiết kế web bất động sản nhà đất và thiết kế website nhà hàng khách sạn để bắt đầu bán hàng trực tuyến hay bán hàng online trên facebook
 
Last edited:
You must log in or register to reply here.
Back
Top