By John Park, Founder of Feedmil

As the web continues to grow on an unprecedented scale, the way people consume information is evolving too. Content is increasingly being delivered in real-time streams, and it is now common to discover information by subscribing to RSS, following twitter status updates, or reading facebook streams. More recently, real-time search engines that attempt to find results based on what is happening now have emerged as a new means to tap into the various streams of information flowing on the web.

While there are lots of ongoing debates on what the real-time search is, the limitations of existing real-time search engines, and how they will evolve, this article will focus on the key issues that need to be considered when assessing the quality and performance of current generation of real-time search engines.

Jumping into the real-time streams:
Following a popular person in twitter definitely gives some kind of fun, but when it comes to a problem of discovering information of your ongoing interest by following people, there’s a pain too. You will not only receive posts you like, but also have to encounter many not-so-relevant posts: Most tweets are expressing sentiments like “Dinner at Red Lobster was great.” rather than providing useful information. With so many tweets sprouting from the people you follow, it is becoming difficult to keep on top of your favorite topic.

Things get slightly better when you use real-time search services that retrieve the results from microblogs. They are simple to use and have straightforward interface that presents results chronologically. But it then becomes very hard to keep up with this never ending flow of volatile information streams. To make the matter worse, the stream of microblog search results usually contains similar content (see the twitter search results below) and is often full of dittos (like retweets). You are likely to miss the important piece of information you really want to find out unless you keep scrutinizing the streams flying by very carefully.

That is, while this type of monitoring based approach for real-time search does a good job of showing what is going on right now for various events (e.g., Iran election) and what is being said about specific companies, it has very low signal to noise ratio. So there is a danger of being drowned in the real-time stream due to the information overload when you jump into the stream.

Real-time web is much bigger than twitter:
Twitter sphere is big, and growing rapidly, but as indicated in the recent survey by sysomos (http://sysomos.com/insidetwitter/mostactiveusers/#most-followers), only 5% of twitter users accounts for 75% of all activity, one quarter of all tweets are generated by bots, and users are quite biased in terms of demographics. Yet, many current real-time search engines only look for microblogs such as twitter and friendfeed, and some only search posts that contain links for the purpose of providing good filtering.

In fact, microblog sphere is still quite short of representing what is happening on the web, since there are a lot more other non-microblog sources out there that generate streams. These include blogs, news, podcasts, and social media posts such as youtube uploads, digg submissions and delicious bookmarks. For instance, the recent tragic death of Michael Jackson was first reported by entertainment site TMZ.com not by tweets.

Considering the facts that many breaking news are still first released from public and social media, not from microblogs and also that certain deeper stories require research and investigation that only professionals can do, it is worth trying search engines that retrieve results from broader sources, not limited to microblogs.

Recency alone is not enough:
With so many live posts, photos, videos, and tweets generated on the fly for a current hot topic, what would you prefer to read first? This question is related to the problem of ranking the real-time search results. Currently most real-time search engines do this by simply matching query with strings appeared in the microblog posts and putting the most recent results at top. Then they keep pushing them down in a flowing river of streams as new results that match the query arrive.

The downside of this approach is that the top ranked entries may not be topically relevant and sometimes they turn out to be spams. Furthermore, there is a high possibility that the top ranked results are quite redundant as they are talking about the same thing.

As a solution to this problem, quality assessment result through identifying most popular, most authoritative, most linked-to, or most re-tweeted items can be taken into consideration. But, such kind of filtering will require further processing that will decrease the freshness of information. An effective alternative then would be to present the topically relevant search results from reliable, quality sources in a timely manner. Nevertheless, balancing the tension among recency, relevance, and quality is not an easy problem.

Keeping sync with web in seconds:
Real-time search engines are expected to produce results on a given query from various sources as they find them. Accordingly, one of the biggest challenges in real time search is to minimize latency – the time lag between creation of some data and getting them indexed.

In order to achieve this, existing real-time search engines are already employing in-memory indexing instead of hard disk, and distributing indexes across multiple machines for parallelizing indexing processes.
On the technology side, traditional pull-style RSS and Atom feeds will be increasingly replaced by push oriented protocols such as XMPP and pubsubhubbub, making it possible to instantly notify new content to interested parties.

Therefore, as the cost of CPU and memory is getting cheaper, the critical issue is likely to become a matter of hardware investment rather than software technology, and economically viable real-time search engine services that can sync with web in the scale of seconds are likely to emerge.

Feedmil: a new real-time search engine

In an attempt to effectively address the issues raised above, we have developed a new real-time feed search engine, feedmil.com that can find topic focused quality streaming sources as well as current hot streams for a topical query. Compared to the other existing real-time search engines, it has broadest possible coverage, encompassing all sorts of streams including microblogs, blogs, podcasts, and public & social media.

As for the ranking of real-time streams, feedmil takes a different approach to go beyond the simple buzz monitoring tool, and balances users’ needs to see results in real-time with necessity to discover information from spam-free, quality sources. As shown in the following screen shot, feedmil retrieves quality sources instead of individual posts, and combines them with a column of search results ranked by recency.

That is, search results are grouped by their streaming sources in feedmil. By doing so, users can find right sources for real-time information and interested users can further subscribe to or follow them so that real-time streams can be directly pushed to the users.

In addition, the popularity filter provided at feedmil allows users to quickly explore streaming sources by specifying whether they want the most popular or more surprising results. In this way, users will have a better chance to find feeds that are not in mainstream and for niche subjects.

Feedmil’s goal is to help people search real-time information of their instant as well as ongoing interest in the most efficient and easiest possible way, enabling them keep track of topics they care about.
Although in its infancy, real-time search is one of the hottest areas in search industry right now, and there appear to be lots of challenges ahead to fully realize its potential. Feedmil is not there yet, but we believe that every new development at feedmil.com is a step in that direction.

So, what’s your favorite real-time search engine?