The concepts of search engines and spam go hand in hand. As soon as the first was invented, the second followed, as sure as night follows day.
What exactly is search engine spam? We can all recognise useless websites when we see them, sometimes filled with machine-generated text and thousands of links. Yahoo defines search engine spam as “pages created deliberately to trick the search engine into offering inappropriate, redundant, or poor-quality search results.”
There is a simple reason why spam websites exist, but it’s not what you think. It’s not simply that someone is always trying to make a fast buck, although that’s certainly true. If Google and Bing were genuinely able to remove spam from their results, there would be no reason for spammers to create spam. Google certainly tries hard, but the spammers are always a step ahead. The reason spam websites can generate money for their owners is the principle of Authority.
Allow me to explain what I mean with a little digression. Authority means trust. It means faith. In medieval times the prevailing wisdom was that all major questions of interest to humanity had already been answered. The answers were to be found in the Bible or in the proclamations of the ancient philosophers, such as Aristotle and Plato. These were authoritative sources and could not be questioned.
The Enlightenment changed all that. It challenged the notion of Authority and instead promoted rational thought and criticism as means of answering questions. The result was a flourishing of the arts and sciences that led directly to our modern world.
How is this relevant to Google? The Google search engine has no understanding of the content of a website. If a website says that black is white, Google has no way of knowing if that is true. The way Google and Bing work is based on the idea of Authority. Search engines identify websites that they consider to be authorities and hubs, and use these to decide which websites are to be trusted and which are spam. Some of the time that works, but often it doesn’t. As a philosophical approach to determining the truth, it’s characteristic of Dark Age thinking.
To gain some insight into how Google works, try searching for “search engine” in Google. You might be surprised to find that Google is listed some way down in the results. Sites like Dogpile and DuckDuckGo are listed ahead of Google. I am not kidding. And at the number-one position is Wikipedia.
That’s right. Google lurves Wikipedia. It tends to show Wikipedia pages as the top result for so many searches these days that it’s almost fulfilled its own prophecy of making Wikipedia the number-one search engine.
Yet Wikipedia is easily manipulated. Anyone can create or modify a Wiki page. I’ve updated entries myself, so I know how easy it is. You don’t even have to be registered. As a journalist, I would never trust a Wikipedia entry unless it tallied with my own prior knowledge, made sense, and could be verified independently. But Google can carry out none of these checks. It has no choice but to have blind faith in authority. And, as the pre-Enlightenment age demonstrated with its witch hunts, inquisitions and crusades, authorities can easily make big blunders.
Search engines based on the absolute power of authority will always make mistakes. They will serve up spam. They will list Wikipedia as the top result for “search engine”. One day, new kinds of search engines will emerge, based on understanding and questioning the meaning of web pages. And when that happens, Google’s prediction of not being the most important search engine in the world may well become true.