What do you mean by a crawler?

A Web crawler, sometimes called a spider, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering). Web search engines and some other sites use Web crawling or spidering software to update their web content or indices of others sites' web content.
 
rishabjoy said:
What do you mean by a crawler?
Click to expand...

A crawler is an automated program or script that systematically examines the World Wide Web to provide updated data for a particular search engine. Although search engine crawlers use many different names, such as web spiders and automatic indexing, the crawling task of the search engine remains the same. The web crawl includes a set of URLs for websites you should visit, called seeds, and then the search engine crawler visits each web page and identifies all the hyperlinks on the page, and adds them to the list of places to crawl. The URLs in this list are periodically reviewed in accordance with the applicable search engine policies. Search engine policies may be different for each search engine, and may be a warning to ensure that some pages that have been added to the index before have not been converted into spam.
 
A search engine crawler is an automated script that browses the World Wide Web in a methodical manner in order to provide up to date data to the particular search engine. The process of web crawling involves a set of website URLs that need to be visited, called seeds, and then the search engine crawler visits each web page and identifies all the hyperlinks on the page, adding them to the list of places to crawl.
 
A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot." it is a program or automated script which browses the World Wide Web in a methodical, automated manner. This process is called Web crawling or spidering. Many legitimate sites, in particular search engines, use spidering as a means of providing up-to-date data.
 
Web Crawlers (Spiders)
(spiders, crawlers, ants, automatic indexers, bots, worms, Web spider or Web robot)

No..not the 8-legged variety. Think of a spider as an automated computer program that systematically searches the web for current and unique updates. The thing you need to remember about the internet is that it is as vast and deep as the ocean (maybe more so). And like the ocean it it gigantic, dynamic and rapidly changing. Therefore, spiders are programmed to work in systematic ways, similar to a filter only choosing items that match certain criteria.

If you are interested in the complexities of spiders there is plenty of reading to be done. However, for the most part all you need to know at a basic level is that they are out there searching the web looking for relevant and unique information and there are ways that YOU can attract them to your site:

1. Spiders love websites with CONTENT:
Be sure your website is well nourished with text. This does not mean repeating yourself, but it does mean fully explaining yourself to the viewer.

Example:
"Our site offers patterns"
versus
"Our site offers patterns of all shapes and sizes for your quilting, stained glass, applique and coloring needs".

2. Too many images are dis-tasteful to a spiders fine palate.
Too much of anything will make anyone feel ill. So,if your site only has images, the spiders probably will pass you by. Cut down on your images, and be sure to use text nearby if you are going to use images.

3. Spiders can be somewhat snobbish...
Get yourself linked to a site that is already in search results. Spiders will go through these websites to find new links (which could be yours!)

4. Hold your site together
Spiders move from one of your web pages through the links to your other web pages, so make sure all your important pages can be reached from wherever you are.

5. Again...text is a spiders favorite flavor.
If you have a button or image pointing to a different page, make it a text link and have one keyword that relates to your business.

6. Have an appetizing URL
Meaning...avoid symbols (ex:?, $, %) in your domain name. Spiders can get easily confused by this and will not even consider your site.
 
A web crawler is an automated search engine program that browses the websites for the purpose of indexing and ranking them on the search engine result page, on every relevant search, on the basis of it's on-page and off-page factors. Web crawler can thus be considered as the examiner appointed on behalf of the search engine that scrutinizes ad ranks your website on the basis of its performance on-page and off-page.
 
You must log in or register to reply here.
Back
Top