What is the main work of Search Engine Spiders?

A program that automatically fetches Web pages. Spiders are used to feed pages to search engines. It's called a spider because it crawls over the Web. Another term for these programs is webcrawler. Because most Web pages contain links to other pages, a spider can start almost anywhere.
 
Spider is a program created by Google, its works is visit to the websites and read their pages and many information. Spiders are typically programmed to visit sites that have been submitted by their owners as new or updated.Entire sites or specific pages can be selectively visited and indexed. Spiders are called spiders because they usually visit many sites in parallel at the same time.Spiders can crawl through a site's pages in several ways. One way is to follow all the hypertext links in each page until all the pages have been read.
 
Back
Top