How does a search engine work?

People just coming into the world of digital marketing, may not fully understand the world of search engines and how they work. This post will help you with the answers.

What is a search engine?
It is a software program that searches for sites on the world wide web based on the words that you designate as search terms.

Dominating the search landscape

Spiders
Search engines need to crawl the web pages and they use the best path which is links. The programs that crawl the web are called “bots” or “spiders” and they index the pages. There are three types of spiders, the Googlebot (Google), Slurp (Yahoo!) and MSNBot (Bing). When these spiders craw the web, they add data to the search engine index. When a user then types in a query, for example “mobile phones”, this is then checked against the search engine’s index of all the web pages it has analyzed. The results that are returned are ranked in order with the best results at the top.

Where do the spiders start?
They start with a list of page URLs that have previously been added to their index. When it visits these pages if it finds new pages, through the links, it adds more to the index. It returns to the sites in the index regularly but no search engine will tell us how often this is. Adding a sitemap to your Google webmaster tools informs Google of your site. A sitemap is an easy way for you to submit all your URLs to the Google index.

Sometimes it takes a while for new pages or any modified pages for the spider to find and index. A web page may have been “spidered” but not yet “indexed.” Until a page is indexed which means it has been added to the index, the spidered pages will not be available to those searching with the search engine.

How does the search engine determine rank a page?
There are billions of web pages on the net and therefore it is very important to make sure your site is featured in the search engine result pages (SERPS). Each search engine has developed a set of rules and mathematical equations, called an algorithm to set the order of its rankings. Unfortunately no one knows what exactly goes into the algorithm. The seo (search engine optimisation) consultant will work with you to help improve your rankings and therefore traffic by following the on page and off page rules about seo.

The algorithm between the search engines is different which is what they give different results for the same query. It is important to not just focus on Google but make sure you rank well on the other search engines.

News publishers and the robots.txt file

Ok so this week, for those that missed it, there was some news regarding google indexing news publishers sites.
We all know that if you do not want google to index your website, you simply write in the code:

User-agent: *
Disallow: /

But for some reason the news publishers think that all their content is going to be indexed by the search engine. According to other blogs, news publishers want to charge google for access to their sites. They want google to pay them to index their site. But of course this will not happen. They know they can block the crawler easily.

Search engines have always checked for permissions before crawling through pages from a web site. Webmasters, including news publishers, are aware and use the Robots Exclusion Protocol (REP) to tell search engines whether or not their sites, or a web page, can be crawled.

Links

The importance of links.

It is very important to have good quality links coming to the site.  Links are essential in SEO, bots and crawlers check the incoming and outgoing links to our site.  The crawlers rank how important this links are and if they are relevant to your site.
There are lots of directories and blog articles where the website can be submitted to. With genuine content, you will be able to generate links with other website and blohs. If you link to other sites then they will link back to us and our page rank number will increase.

Link Building methods
Links should be built from a variety of places
-    Web directories such as dmoz.org or business.come
-    Seo press releases
-    Blogs and blog comments
-    Any other of your other websites (eg parent company)
-    Social booking like de.licio.us, Digg
-    Niche websites in our area
-    Affiliates

•    Search Engine and Directory Submissions
This is an important step in link building process.
You should not submit to all search engines, this is a timely and costly process (some require a submission fee).  Alternatively, you could submit the site to webdirectories – global and local depending on your target audience.
You should submit your site to the Yahoo directory and the open directory http://www.dmoz.org/. Once submitted, the search engines should fine your site within 2 or 3 months.
Alternative free directories
www.splut.com (directory of UK websites)
www.thefreewebsitedirectory.com
www.worldwidewub.org
www.wowdirectory.com
•    Online press releases
Press releases can also be an effective way of promoting your website and increasing traffic and awareness of the site. This is also a good way to build inbound links to the online community.
•    Blogging
Blogging is a very important way to upload fresh content to the website
It also provides an area where you or your company can communite with its customers.  There are a lot of free blogging software.  One that I am using now is wordpress!!
•    Social media
This is a growing factor in helping promote websites, facebook, linked in and social bookmarketing sites such as de.li.cious, digg and stumbleupon. These sites can help in building back links, articles about your site and as a result increasing traffic to the site.
•    Internal Site Linking

This is also an important aspect of link building but is often overlooked.  Internal keyword navigation is an important party of search engine algorithm.  Some sites may have very few internal links.
•    Link scouting
It is important to research sites that could provide a potential link to your website.  This means a lot of work should be done into the competitor link campaign and web searches to find potential sites for us.

I bet you didn’t think I could write some much about link building!