How the Search Engines work

With Google being by far the most visited website in the world, it is hard to imagine using the Internet without search engines to help you find the content or businesses that you are looking for. Google, Bing, Yahoo and other search engines have two main roles. Firstly, they use robots to crawl the Web and build an index of websites and their individual pages. Secondly, they provide answers when someone enters a query. Gaining a better understanding of how the search engines work is important for any online marketing campaign, since it is necessary to capitalize on the fact that most Web traffic is driven by the search engines.

Given the fact that search engine traffic is so important for any website, online marketers use a process known as search engine optimization (SEO) to increase the visibility of their websites in the search results. SEO is a part of the greater discipline of search engine marketing (SEM), an area which also incorporates paid search engine advertising in the form of sponsored links at the top of the search results. By contrast, SEO doesn’t cost anything, instead delivering organic traffic to your website. Whether you use sponsored advertising or not, the majority of your traffic will be organic.


Crawling and Indexing the Web

  There are around a quarter of a billion active domains (individual websites) on the Internet, and this number has been growing exponentially every year ever since the very first website was launched in 1991. Each website consists of one or more individual webpages (URLs), with many websites consisting of hundreds or even thousands of different pages. The virtual world is simply vast, making it practically impossible to even grasp the enormity of the amount of content out there.


With the size of the Internet comes the immense challenge of formulating a directory containing information about almost every webpage out there. There simply aren’t enough people on the planet to create an index of all of the content on the modern Web, so we have to use robots instead. These search engine robots, also known as spiders or crawlers, work tirelessly to scan the Internet and index its content.


Aside from actual content, the Internet is also a vast network of links, connecting everything together. The crawlers follow these links to create a map of the Web while indexing and ranking content in the process, and the information collected is stored in vast datacentres. These datacentres, also known as server farms, contain many thousands of computers and countless hard disks with enough capacity to store the thousands of terabytes of data collected.


Delivering Search Results

  Google receives, on average, 5 billion search queries per day or around 60,000 per second, and search results are typically delivered within a second or two. When you enter a search query in Google, or any other search engine for that matter, the software has to scan through the index built by the crawlers and find results which relate to the keywords entered. An incredible amount of work takes place every time anyone enters a query, and the computing power required to deliver relevant results is quite unimaginable.


The goal of the search engines is to return results which are relevant to the queries entered, and the results are ranked in terms of relevance in the process. The relation between your own website and the search engine results revolves around the keywords people are most likely to enter when searching for a website such as your own.


When the Web was young, search engines were far more simplistic than they are today. Although the basic process involves finding pages containing keywords matching the queries entered, numerous other factors now come into play so that the results are not only relevant – they are also trustworthy, informative, useful and high in quality.

In fact, Google’s algorithm now uses around 200 ranking factors when determining a webpage’s place in the search results.