Dr. Christian Maaß

Original Article: Soziale Suchdienste und algorithmenbasierte Suchmaschinen im Vergleich

In recent times it is striking that search engines based on alternative approaches are entering the market increasingly. Social search approaches take over a special role within this context. They are available taking into account several characteristics and – in contrast to search engines as Google, Yahoo and Ask – involve the user when generating the index. Up to now, the scientific discussion on this topic is still in its infancy. In the context of a project in cooperation with Prof. Dr. Dirk Lewandowski we have deducted a seminar on the topic of social search approaches. Amongst others, one topic was to compare the relevancy of hits of algorithm-based search engines with social bookmarking systems and question-answer-services. In this place, I want to shortly introduce some of the results, a more detailed version of the report will appear as edited volume in some weeks from now.
To find an answer to the question in how far social search approaches can compete with Google & Co, a test on relevancy has been carried out, taking into account the following three types of search approaches:
· Google, Yahoo and MSN have been chosen as samples for algorithm-based search engines, as they are the three largest providers worldwide maintaining an own index
· The best known German-speaking and English-speaking social bookmarking systems have been included in the analysis by picking Mister Wong and del.icio.us
· Finally, the question-/answer-services LycosIQ and Yahoo! Clever have been chosen to form part of the analysis, which have to be seen as a special manifestation of social search approaches
More than 50 search inquiries have been set for each of the services listed above so that the error probability is minimized. To analyze the relevancy of results generated by algorithm-based search engines and social bookmarking systems, the first ten results are taken into consideration, the analysis of question-answer-services is based on the first three questions displayed. The results of these inquiries have been presented to a group of judges anonymously. The judges neither knew the rank of the search result nor by which search engine the hits were generated. Criteria to be taken into account were
· Precision of search results,
· Intelligibility of search results and
· Trustworthiness of search results.
Precision describes the percental number of relevant search results within the total number of results generated. Focusing on the analysis based on the criteria “medium precision” (which includes all judgements of one or two on a scale from one to six, one being most relevant), it turns out that Google comes off best with a Precision of 46 Percent (cf. Figure 1). Google was able to deliver 15% more relevant search results than MSN. If one compares algorithm-based search engines to social bookmarking services and answer-/question-services, the latter are able to generate a higher degree of relevant results. Within this group, the performance of Lycos IQ is better than the performance of Yahoo! Clever. The worst results are generated by social bookmarking systems, which can provide less than half of the amount of relevant hits than Google. This can be lead back to the immense number of defect links and obvious problems with maintaining the indexes up to date. Furthermore, especially del.icio.us reveals a lot of links to pages in foreign languages which may be relevant, but cannot be understood. But it has to be stressed, that even the first ten results of the “top performer” Google deliver at least 5 results which are not significantly relevant, so that the results cannot be regarded as satisfying in the end.

Figure 1: Precision of search results

Regarding the intelligibility of search results, 84% of the hits within the sample are understandable. Bookmarking systems provide the highest amount of results which cannot be understood. In this case it is especially noticeable that the judges estimated 25 percent of all results generated by del.icio.us as not intelligible (cf. Figure 2). All other search engines provide a relatively low amount of results which can be understood.

Figure 2: Intelligibility of search results

Furthermore it was tested, in how far the search results can be taken as trustworthy. Here, too, Google & Co fare better than social bookmarking systems (cf. Figure 3). Question-/Answer-Services have not been included in this analysis, as the criteria needed have not been prompted.
Only 40 percent of the results of social bookmarking systems are judged on as trustworthy. This differs from the results expected in advance, as the content and the links provided are exclusively chosen by users. Another phenomenon regarding the trustworthiness is that Wikipedia-pages are regarded as very trustworthy in spite of just containing user generated content. Overall, 5% of the hits in the sample provide Wikipedia-links, 63 percent of which are regarded as trustworthy.

Abb. 3: Trustworthiness of search results

The results, which are only sketched in this place, reveal that social search services can not compete with algorithm-based search engines at the current point of time. But it has to be added that social bookmarking systems have been designed for private link collection, not as search engine. But what is more important: There are a lot of activities and projects that are on a good way to establish alternatives to global players like Google, Yahoo and MSN.

This entry was posted on Sunday, March 30th, 2008 at 9:00 am and is filed under Guest Authors. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.