What is the difference between Robots.txt and Spider?

[mjhonson]

Friends...Help me to Understand these Crawling ?

 

[sadianisar]

Spider: It is basically known as web crawler. It visits website or webpages or links etc and then crawls to index it and save it in its own database.

Robots.txt: It is a file in which we write commands to instruct crawlers to not visit those webpage or part of the website which we do not want it to..

 

[mjhonson]

thank you for your explaining

 

[edwina]

Both are same.
They are programs which search engines use to crawl web pages for indexing.

 

[mjhonson]

Both are same.
They are programs which search engines use to crawl web pages for indexing.

but on the other side on robots.txt we commands to instruct crawlers not to visit those webpages or part of the website which we don't want it to index for crawling..

 

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Spider is a search engine automated program that are responsible to read through webpage source and provide information to search engines.

 

[Sonika11Mehra]

Spider:- Search engine spider is Crawler which crawls the website page and index your website in the search engines.

Robot.txt :- It the file place in your website through which the crawler is stopped to website to index only that part which you want to index.

 

[seomarqetrix]

Spider, crawler , and robots both are the same program to reads the text information from World Wide Web and find hyperlinks if any, then followed those links and collect all data and records it into search engine database for the use of visitors.