Search and the Quest for Order of Magnitude Suggestion

Over the last couple of years, those of us interested in search have essentially “turned over the same stones” ten thousand times to grasp the course and future of search engine technology. Semantics, AI, computational algorithms, social search, suggestion engines and a host of other methodologies have emerged as components of the race for search relevance. Perhaps this race should be more correctly tagged “the race for perceived relevance toward alternative search engine marketing”? I think what we are really seeing in the search race of late is something akin to tagging organizational methodologies as natural language or semantic search.

In many conversations with leading search developers like Riza Berkan of hakia and Barney Pell of Powerset, several recurring themes emerged. Initially these “semantic” technologies were set against Google in a sort of “David and Goliath” competition for what appeared to be an actual search for quality and relevance. Knowing the two individuals mentioned and others fairly well, I am sure that as scientists their primary motivation was just that. However, once backing and dollars entered into the equation, obviously the focus shifted more and more toward the monetization aspects of this race. This is not to say that financial motivation precludes quality necessarily, but the impact can be quite powerful and does not always lead to the “actual” destination desired.

 Courtesy Silicon Valley Insider – Google supplants Windows soon

This is no big secret, but if we think about what has been presented to us as a race for relevance, then we must consider the degree and character of even the term relevance. This is a rather nebulous idea, but consider that regardless of the technology employed –human preference and control actually maintain the engine. What we may have overlooked in the search game is this human element. In the end, we are so often subjected to human suggestion even when derived from mechanical intervention. Search is and may always be about suggestion.

Human Beings, Semantics and Possibility

The dedication and intellect behind these search engines are undeniable and I have always been a supporter of the “possible” whether the methodology were hakia’s, Powerset’s, Search Wikia’s or even Google’s. This semantic question at first seemed “addressed” early on by Berkan and Pell, though the methods employed were obviously quite different. Semantic search from an empirical point of view proposes one huge problem for any of the latest innovations – practical, exacting possibility – though many will disagree. Whether formal semantic theory or even cognitive semantic regimen underlie the philosophies of the developments, true semantic search relevance without almost super human intervention is impossible today, but the closest approximation is not.

Impossibility is not a term I have ever really used much, but consider the “exactitude” of what we have been talking about. Outside “the long tail” and other incubations of the discourse about search, there are galaxies of disparate problems inherent in both meaning and its communication. Semantics by definition is either about “literal” meaning or either about pragmatic meaning that is so diffused as to be useless in search. Of course there is a lot of philosophy in between, but now we are talking about conciliation. Certainly Google and all others are already in this compromising mode with regard to relevance in the strict sense. Outside self aware artificial intellect (which I know hakia has been exploring), a refined compromise is all we can expect with current technology in my opinion.

Best Guesses and Presentation

To anyone tuned in to this search battle one aspect is readily visible – outside the determined quest for better results the up and coming engines have attacked Google at its weakest points – organization and presentation. A great timeline and introduction to Powerset’s new organizational features was written by Charles a little while back at the onset of their release. Loren Baker and I talked with Barney Pell of Powerset when they announced their latest breakthrough before the Microsoft takeover. You can derive a lot out of both the discourse and the graphical elements of these articles as far as what we might call “the progressive improvement of Powerset via hard competition”.

 Organization Web 3.0 Style

 

Note the similarity of organizational function – Competitive refinement?

The Pell conversation led me to two observations from the relative “back and forth” competition between Powerset and hakia. Firstly, Barney and company had made a compelling re-organization in the way search results are presented (a sort of organization of organization). Secondly, Barney readily and very honestly admitted that this refined way of organizing, utilizing and presenting search was what he termed “baby steps” toward true natural language search. I was amazed and actually a little astonished at both the logic of Powerset’s delivery and at the hammer blow of honesty.

What this meant essentially, was exactly what I am contending in this article – even our best and most dedicated minds cannot actually produce more than “baby steps” toward the presupposed goal of true semantic search. Another indicator of merit in this search race has been the elevation of the search game out of an aspect of competition. Hakia makes one move, Google refines its pseudo – semantic aspect, and Powerset (behind the doors) refines its organization presentation. The use of Wikipedia as a “compartmentalized” search index in retrospect was both a stroke of genius on the part of Pell and company and perhaps an indicator of things to come.

The Human Element Revisited

All of us interested in search have essentially bounced around somewhat driven by possibility. We write about the innovation, the people behind the technology, the business of search and we really are enamored with whichever engine or entity presents, well – hope. Many of us are driven by the hope that someone will produce something as important as answers to our most desired or needed questions. Being Web enthusiasts and doing most of our business here, there is also the efficiency and news aspect to this “enthusiasm”. There is one entity, one element as I mentioned earlier, that we have come across and often overlooked frequently – the power of human beings. Enter and re-emerge Search Wikia.

A couple of weeks ago a great friend, Angela Beesley, co-founder of Wikia with Jimmy Wales, sent me a link to download the new Search Wikia toolbar. To be honest I had not been apprised of the development there in some time, and when I downloaded the tool I was stricken. The sheer power of what this tool might allow to be accomplished combined with what we now know of organizing and indexing search ala hakia and Powerset, made me have one of those “Aha” moments. I remember asking Pell, Berkan and Jimmy Wales about the logic and prospects of aligning very powerful semantic search with the human filtering capability of Search Wikia (also known early on as Wikiasari). To a man, none ruled out the potential or possibility. But on thinking of the alignments and relative developments I cannot help but analyze where search stands in light of a new definition of “semantic search”.

Given the technology exists, but is curtailed by the human end of the long tail, our refinement of this “search compromise” can only reach its most viable end via human suggestion. What Search Wikia is doing by way of supplying a tool for refining results is clear now. Whether Jimmy Wales and company align with Microsoft of go it alone, it is apparent that given time and patience, what Search Wikia will accomplish in the end, will be at least as popular or relevant as any other search engine we have mentioned. Why? Because all these engines are suggestive and suggestive only.

The All Seeing Eye

Suggestion. Google, Yahoo!, hakia, Powerset (Microsoft), Mahalo or any engine ever devised simply “suggests” relevance based on variables. There is no disputing this mathematically or intrinsically. So, the question remains; “How do we determine relative relevance, and who do we trust to provide it?” Google came to prominence because the tech world (geeks) wanted something akin to hot keying their way to Web dominance. The brand and the utter simplicity of Google appealed first to the solidified group predominant on the Web, while forces behind the scenes vied for position (sometimes clumsily in the case of MS and Yahoo!).

The tiring and inherent ugliness of Google – compare

All the while search engine marketing has been a huge factor even if obscured by our love affair with technology. A great article by Brad Howarth dealing with the divide in methods of Web 2.0 philosophies addresses marketing motivation interestingly. So, given the “not so obscure” nature of goings on under our eyes, consider another entity – Jimmy Wales. Take a little time and consider just how humbly dominant Jimmy Wales’ endeavors have been on the Web. If you wanted to approach a type of dominance on the Web, how might a very smart man accomplish this?

Microsoft or Google now, might simply utilize their power to perpetuate the “snowball” of progress in a rather high profile methodology as we have seen. However, if you use any search engine out there and type in virtually any word, within the top 5 results there will be a Wikipedia entry. Not only will there be one, but given the relevance of others it is probably the most relevant too unless you are searching for a commodity.

You may ask; “What does this mean?” It means, that regardless of how imperfect Wikipedia may be, the relevance and content there surpasses as a whole, anything on the Web unless the Library of Congress goes live. What’s more, the all seeing eyes that watch over the entries are not doing too badly. Remember, everything we encounter is an opinion regardless of the empirical back end…even mathematics. It may also mean that Wales being a patient and relatively low profile developer had a plan all along. Time and pressure, and perhaps with the answer in hand to the search game? Indexing the Web from the inside out!

Short Tail of a Long Story

Conspiracy theories aside, I am not a believer in coincidence. It is not coincidental that Powerset chose to use Wikipedia as a matrix for their development, it is not a coincidence that Microsoft bought them and it is decidedly no coincidence that Wikipedia is one of the underlying and perhaps most pervasive factors on our Internet. I know Wales fairly well, and besides being a nice guy, he is extraordinarily astute.

If search and relevance can be boiled down to either mathematical or human preferential suggestion, then each of the leading contenders for market share have advantages. Google is powered primarily by popularity, technology and to a great extent derived data. Microsoft obviously has the financial and technical capability to compete, and given the acquisition of Powerset – coolness and a head start on new age result organization. Search Wikia has a user base and extensive experience in the social search or human powered search potentiality. Perhaps in the end, this might be the biggest advantage of all, given that it would take some catching up on the part of the others to derive the same advantage.

The search for the best search will obviously never end. Google will not be toppled any time soon, but I expect t this is not the goal of any of the contenders. Rather, these entities are after that huge lead Google has in market share. Google in its current position is just not an acceptable outcome for the rest of the world intent on any interjection on the Web.

The Best Engine Ever

As for the best search engine ever devised (I will give this one to the engines for free) it would be something like Powerset, with Google’s advantage, overlapped with a social search aspect like Search Wikia. To make it really unique and worthy, it should allow for toggling the social or even AI aspect on or off to automatically reset the results for relevance. Imagine this ability with Google – Poweset or hakia organization – and Search Wikia refined human suggestion. As for what is to come, make no mistake, the big players are driven by time and pressure as well. Refined search is like anything else, we can make it as complicated or simple as we like, but in the end we cannot get past the eternal question; “If people have an impossible time understanding one another, how can a machine hope to do better?” Every book you ever read, blog entry ever made, or even your smartest professor’s discourse was nothing more than an opinion or suggestion of one. Sure, some empirical data may have been employed, but none the less – suggested semantics is what we get. So this opinion, like so many others will only be substantiated with time – or perhaps not. You choose what is relevant, your power to understand is much more powerful than Google.

Ideas power the Web, what do you think“they“ are thinking?

Courtesy Phil Butler

This entry was posted on Sunday, September 7th, 2008 at 8:00 am and is filed under Guest Authors. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.