How Search Engines Work ?

First, let’s look at how crawler-based search engines work (both Google and Yahoo fall in this category). Each search engine has its own automated program called a “web spider” or “web crawler” that crawls the web. The main purpose of the spider is to crawl web pages, read and collect the content, and follow the links (both internal and external). The spider then deposits the information collected into the search engine’s database called the index.