How does a search engine work?

People just coming into the world of digital marketing, may not fully understand the world of search engines and how they work. This post will help you with the answers.

What is a search engine?
It is a software program that searches for sites on the world wide web based on the words that you designate as search terms.

Dominating the search landscape

Spiders
Search engines need to crawl the web pages and they use the best path which is links. The programs that crawl the web are called “bots” or “spiders” and they index the pages. There are three types of spiders, the Googlebot (Google), Slurp (Yahoo!) and MSNBot (Bing). When these spiders craw the web, they add data to the search engine index. When a user then types in a query, for example “mobile phones”, this is then checked against the search engine’s index of all the web pages it has analyzed. The results that are returned are ranked in order with the best results at the top.

Where do the spiders start?
They start with a list of page URLs that have previously been added to their index. When it visits these pages if it finds new pages, through the links, it adds more to the index. It returns to the sites in the index regularly but no search engine will tell us how often this is. Adding a sitemap to your Google webmaster tools informs Google of your site. A sitemap is an easy way for you to submit all your URLs to the Google index.

Sometimes it takes a while for new pages or any modified pages for the spider to find and index. A web page may have been “spidered” but not yet “indexed.” Until a page is indexed which means it has been added to the index, the spidered pages will not be available to those searching with the search engine.

How does the search engine determine rank a page?
There are billions of web pages on the net and therefore it is very important to make sure your site is featured in the search engine result pages (SERPS). Each search engine has developed a set of rules and mathematical equations, called an algorithm to set the order of its rankings. Unfortunately no one knows what exactly goes into the algorithm. The seo (search engine optimisation) consultant will work with you to help improve your rankings and therefore traffic by following the on page and off page rules about seo.

The algorithm between the search engines is different which is what they give different results for the same query. It is important to not just focus on Google but make sure you rank well on the other search engines.

Google Instant Results

Google have launched a new search enhancement feature called “Instant”, this is how they describe it:

“Google Instant is a new search enhancement that shows results as you type. We are pushing the limits of our technology and infrastructure to help you get better search results, faster. Our key technical insight was that people type slowly, but read quickly, typically taking 300 milliseconds between keystrokes, but only 30 milliseconds (a tenth of the time!) to glance at another part of the page. This means that you can scan a results page while you type.”

Google Instant Results

The next question most people are concerned with is, how will Google Instant affect seo?

Google has again changed the personalised results and is now affecting what the user sees in search – paid and natural. It is something to be aware of, but it will only affect those that are logged into Google.

1. Unique search queries
According to Google, 25% of searches are unique. Therefore if someone is offered new results in the search box, they may stop writing the rest of their search query, potentially lowering the number of unique search queries.

2. Long tail search
There may be a change in the long tail patterns of keywords as people are offered instant results. There may be a decrease in the number of long tail search queries typed into Google.

3. Brilliant basics SEO
Title tags and meta descriptions are even more important as this is what comes up in Google in the instant results. It is important keywords are in the title and description tag, encouraging users to click through.

Google Instant may just be kept as a default but there is an “off” switch and you also have to be logged in to see it. It is certainly distracting when typing in search results into Google. Google has even admitted it does not know yet how users will interact with the search results, ads and search experience. However, one thing is for sure, users should find relevant results quicker than before (Google say 2 – 5 seconds saved per search) and seo and ppc practioners will need to keep an eye on how Google Instant affects traffic to their site. The search landscape is constantly changing and the seo and ppc team are always adapting well to these modifications implementing best practice techniques to their campaigns.

Mobile search news

2009 is the year of mobile search.
Google have developed their offering of mobile search over the past year which covered some key milestones.

- In April they launched Product Search for Android and iPhone
- In July they launched Product Search for other web enabled phones. Product search allows users to read product ratings and reviews while you are shopping helping you to get the best deals.
- In June Google launched a new version of iGoogle for Android and iPhone devices making more of desktop iGoogle gadgets available on the mobile.
- August saw Image Search results optimised for feature phones.
- In September they launched the mobile version of Fast Flip for Android and iPhone, allowing users to scan news articles quickly.
- In October they launched Search Options for Android, iPhone and Palm WebOS phones, allowing users to further refine search results to get the information they are after.

To find more about mobile search visit google’s blog.

I will be adding more shortly.