BACK TO HOW-TO ARTICLES

How Search Engines Work

by Ken McGaffin, 24 November 2006

One of the most frequently asked questions from website owners is, "Why can't my site be found on Google?" They know it's important to appear in search engine results but they just don't know why it doesn't happen to them.

Key points

The Spider software crawls the web looking for new pages to collect and add to the search engine indices. The Index software catches everything the Spider can throw at it.  They may well be in awe of the 'black arts' of search engine optimization or puzzled by the complexity of it all. If they're unlucky they will have paid out money to some snake oil salesman guaranteeing
to get them to the top in 48 hours - and been sorely disappointed with the lack of traffic that results.

Most search engine optimizers are highly ethical, professional people but they do tend to keep their cards close to their chest.

The Big Secret is There is No Big Secret

It's true: the 'big secret' of search engine optimization is that there is no big secret.

It is all about understanding what is going on, followed by the hard work and attention to detail that are common to many business activities.

This article is about understanding what is going on inside a search engine. It is just under 800 words in length but that is more than enough to give you what you need to know, albeit in simple terms.

Inside the Guts of a Search Engine

For simplicity's sake, let's say there are three pieces of software that together make up a search engine - the Spider
software, the Index software and the Query software.

If you understand what these three do, then you have the foundation for getting your website to the top of the search
engines.

Here's what the three types of software do:

The Spider software 'crawls the web looking for new pages to collect and add to the search engine indices'. This is a metaphor. In reality, the spider doesn't do any 'crawling' and doesn't 'visit' any web pages. It requests pages from a website in the same way as Microsoft Explorer, or Firefox or whatever browser you use requests pages to display on your screen.

The difference is that the spider doesn't collect images or formatting - it is only interested in text and links AND the URL,
(for example, http://www.Unique-Resource-Locator.html) from which they come: it doesn't display anything and it gets as much information as it can is the shortest time possible.

A spider loves links because they lead it to other web pages that have the things that it loves, guess what? Text, links and URLs!  The Index software catches everything the Spider can throw at it (yes, that's another metaphor). The index makes sense of the mass of text, links and URLs using what is called an algorithm - a complex mathematical formula that indexes
the words, the pairs of words and so on.

Essentially, an algorithm analyses the pages and links for word combinations and assigns scores that allow the search engine
to judge how important the page (and URL) might be to the person that is searching.  And of course it stores all of this information and makes it available.

The Query software is what you see when you go to a search engine - it is the front end that everybody thinks of as a search engine - familiar ground at last. It may look simple but it presents the results of all the quite remarkable search engine software that works away invisibly on our behalf.  The main feature of the query software is the box into which people type their search terms.

Type in your words, hit search and the search engine will try to match your words with the best web pages in can find through searching the web.  But this too is a metaphor and perhaps the most important one.

The query software doesn't search the web - it checks the records that have been created by its own index software. And
those records have been made possible by the raw material the spider software collects.

What You Need to Understand About Search Engines

That is it. What you need to understand is that the search engine has done all the hard work of collecting and analysing web pages, BUT it only makes that information available when someone does a search by entering words in the search box and hitting return.

The words people use when they search therefore determine the results the search engine presents. So search engine optimizers want to know the words people use when they search - we call them keywords - that might sound fancy but keywords are only 'the words people use when they search'.

And that's what Wordtracker provides. Use keywords in your website copy and you will prosper: ignore them and your online business will surely perish. To try Wordtracker for free just click on the Wordtracker graphic link (2nd from the left) at the bottom of this page.

BACK TO HOW-TO ARTICLES

"Tom continues to work for his clients after there website is established by providing technical support and suggestions to further improve the performance of your website." - Gordon Eberhardt, Owner, Sherlock Homes Inspection, LLC
 read more testimonials