Search and the Quest for Order of Magnitude Suggestion

Over the last couple of years, those of us interested in search have essentially “turned over the same stones” ten thousand times to grasp the course and future of search engine technology. Semantics, AI, computational algorithms, social search, suggestion engines and a host of other methodologies have emerged as components of the race for search relevance. Perhaps this race should be more correctly tagged “the race for perceived relevance toward alternative search engine marketing”? I think what we are really seeing in the search race of late is something akin to tagging organizational methodologies as natural language or semantic search.

In many conversations with leading search developers like Riza Berkan of hakia and Barney Pell of Powerset, several recurring themes emerged. Initially these “semantic” technologies were set against Google in a sort of “David and Goliath” competition for what appeared to be an actual search for quality and relevance. Knowing the two individuals mentioned and others fairly well, I am sure that as scientists their primary motivation was just that. However, once backing and dollars entered into the equation, obviously the focus shifted more and more toward the monetization aspects of this race. This is not to say that financial motivation precludes quality necessarily, but the impact can be quite powerful and does not always lead to the “actual” destination desired.

 Courtesy Silicon Valley Insider – Google supplants Windows soon

This is no big secret, but if we think about what has been presented to us as a race for relevance, then we must consider the degree and character of even the term relevance. This is a rather nebulous idea, but consider that regardless of the technology employed –human preference and control actually maintain the engine. What we may have overlooked in the search game is this human element. In the end, we are so often subjected to human suggestion even when derived from mechanical intervention. Search is and may always be about suggestion.

Human Beings, Semantics and Possibility

The dedication and intellect behind these search engines are undeniable and I have always been a supporter of the “possible” whether the methodology were hakia’s, Powerset’s, Search Wikia’s or even Google’s. This semantic question at first seemed “addressed” early on by Berkan and Pell, though the methods employed were obviously quite different. Semantic search from an empirical point of view proposes one huge problem for any of the latest innovations – practical, exacting possibility – though many will disagree. Whether formal semantic theory or even cognitive semantic regimen underlie the philosophies of the developments, true semantic search relevance without almost super human intervention is impossible today, but the closest approximation is not.

Impossibility is not a term I have ever really used much, but consider the “exactitude” of what we have been talking about. Outside “the long tail” and other incubations of the discourse about search, there are galaxies of disparate problems inherent in both meaning and its communication. Semantics by definition is either about “literal” meaning or either about pragmatic meaning that is so diffused as to be useless in search. Of course there is a lot of philosophy in between, but now we are talking about conciliation. Certainly Google and all others are already in this compromising mode with regard to relevance in the strict sense. Outside self aware artificial intellect (which I know hakia has been exploring), a refined compromise is all we can expect with current technology in my opinion.

Best Guesses and Presentation

To anyone tuned in to this search battle one aspect is readily visible – outside the determined quest for better results the up and coming engines have attacked Google at its weakest points – organization and presentation. A great timeline and introduction to Powerset’s new organizational features was written by Charles a little while back at the onset of their release. Loren Baker and I talked with Barney Pell of Powerset when they announced their latest breakthrough before the Microsoft takeover. You can derive a lot out of both the discourse and the graphical elements of these articles as far as what we might call “the progressive improvement of Powerset via hard competition”.

 Organization Web 3.0 Style

Note the similarity of organizational function – Competitive refinement?

The Pell conversation led me to two observations from the relative “back and forth” competition between Powerset and hakia. Firstly, Barney and company had made a compelling re-organization in the way search results are presented (a sort of organization of organization). Secondly, Barney readily and very honestly admitted that this refined way of organizing, utilizing and presenting search was what he termed “baby steps” toward true natural language search. I was amazed and actually a little astonished at both the logic of Powerset’s delivery and at the hammer blow of honesty.

What this meant essentially, was exactly what I am contending in this article – even our best and most dedicated minds cannot actually produce more than “baby steps” toward the presupposed goal of true semantic search. Another indicator of merit in this search race has been the elevation of the search game out of an aspect of competition. Hakia makes one move, Google refines its pseudo – semantic aspect, and Powerset (behind the doors) refines its organization presentation. The use of Wikipedia as a “compartmentalized” search index in retrospect was both a stroke of genius on the part of Pell and company and perhaps an indicator of things to come.

The Human Element Revisited

Read the rest of this entry »