Is One Search Engine Enough? Not By a Long Shot.

January 13th, 2009 by Guest Author
Posted in Guest Authors | 2 Comments »

kosmix

By Guest Author Digvijay Lamba

I hear it all the time:  the mistaken assumption that one search engine is all anyone needs to find information online.  Only one way to find everything on the Web?  Think again.

Today’s search landscape is far more fragmented than most people realize.   Traditional search engines are only part of the story:   the definition of a search engine has now expanded to include any website with a search box. Think of the way you shop for a book on Amazon, find an Italian restaurant in San Francisco on Yahoo Local, reconnect with your long lost high school pals on Facebook, or research the drug your doctor prescribed on RightHealth.   Search engines are everywhere.

Because these different search engines have few connections to each other, users never see the vast majority of content that would be valuable to them for each query.  No search engine—not even the big three—can surface every bit of useful content and present it in an easy-to-digest way. That’s the bad news.  The good news is that search is going to have its revolution soon, which will completely change how we find information.

How will that revolution happen? The potential for innovation will cause a community to evolve around the search space, and search will become a platform.

The possibilities for innovation in search are huge.  For example, someone will find a better way to let users specify their intent.  Requested information will be returned in a more visual, intuitive layout. Results will be personalized better based on the user’s intent and the subject of the search. User interfaces will be tailored to the specific intent (Looking for videos?) or the more specific subject (Just want Health results?). Domain-specific experts will achieve a deeper understanding of both the content and the query, and will form richer connections between the two.

As the potential for innovation in Search grows, the space will attract a much larger pool of entrepreneurs than any one company can ever contain.  Whenever a technology matures, communities naturally form around it.  Take personal computers, for example.   While ATI and NVidia fight over the Graphics capability, Intel and AMD compete over the chip. Dozens of companies provide monitors, keyboards, memory, motherboards and every other aspect of the PC. And hundreds if not thousands of companies can provide the different software applications.

Every widely-used technology has hundreds of companies involved in competing for the different parts that make up the whole, and the search space will be no different. Expect a flurry of increased funding for specific solutions, increased competition, and increased specialization.  And the company that harnesses this momentum and gets companies engaged together in a search platform will find itself in the most enviable position.

What do we mean by a search platform? It will be an internet website that allows hundreds of search companies to provide specialized solutions to various search problems in a connected and integrated manner. For example, if you search for “Chocolate” the platform may connect you to a recipes search engine, a health search engine, and a shopping search engine allowing each to present specialized results with a richer interface than what today’s web search allows. It will allow each engine to innovate in its area of expertise and connect the three together in a meaningful way so the end user sees much richer results.

The platform creator will have the responsibility to match different search solutions to the user’s differing needs.  What’s the best approach to achieve this?  One option is to let the users choose from the various search providers and “install” the applications that best suit them.  Another approach is to build a platform that understands the user and the different applications deeply, and can connect the user automatically to just the right application, at just the right time.  My company, Kosmix, has built an early version of such a platform, which delivers the best of the Web by bringing together hundreds of content providers, aggregators, and niche search engines. We’re off to a good start, and this is just beginning.

For search to be viable as a platform in the long-term, it must offer value to everyone involved: the platform creator, the application providers, and the users. The platform creator and application providers need to make money, obviously, while at the same time offering consumers high-quality, easily accessible content for free.  This can be achieved in one of three ways:  The Amazon Model, the eBay Model, or the Facebook model.

In the Amazon model, the platform creator owns the underlying economics and is responsible for sharing the benefits with solution providers.  Think of the way Amazon.com receives payments for anything a customer buys on the site and then, in turn, pays the sellers on the Amazon marketplace.

In the eBay model, solution providers own the revenue and share a part of it with the platform creator. eBay does this by charging its sellers a percentage fee for all items sold on the marketplace.

In the Facebook model, the platform creator and the solutions providers are independently responsible for their own benefits. Facebook gets an indirect benefit by making the experience on their site richer for their users, while allowing applications to display ads or generate traffic. In this case, the platform must offer opportunities for the owner to monetize his application in various ways. For example they may share traffic, share the advertising space between the two, etc.

All three of these models have merit, and it remains to be seen which direction a search platform will take. One thing remains clear: as search evolves it will become harder and harder for one company to do it all. A platform that connects hundreds of search engines together can become a powerful source of innovation allowing it to build a deeper and richer experience for its users. The innovation in search has only just started. We need millions of connected search engines.

Search for Scientific Books with Dandelon

January 13th, 2009 by Guest Author
Posted in Global, Guest Authors | 1 Comment »

dandelon

By Guest Author Manfred Hauer

Dandelon.com is a public search engine for scientific books and articles. The user interface supports 27 languages and user queries are expanded by multiple thesauri in up to 25 languages. The content is a combination of bibliographic data generated by libraries, automatic linguistic indexing and table of contents and some whole texts of articles or books. The combination of scanned tables of content, broad linguistic indexing, extensive use of multilingual thesauri (synonyms, translations and narrower terms – all written or imported into IC INDEX) and relevance ranking opens a new level of library searching. The 1.7 million thesaurus entries are visible as a Flash application in the “dictionaries” section.

The collection is currently growing by about 2500 books and 2000 articles per week; in January 2009 there are about 550.000 books and 600.000 articles. In 2008, 750.000 queries were typed in at first page.

Originally “dandelon.com” was designed for demonstrating the power of “intelligentCAPTURE mobile”, which is a high integrated station for scanning table of contents, articles etc., doing OCR and machine indexing
and communicates with major library management systems. It also captures ONIX, MARC and SWETS data provided by publishers and periodical agencies.

Some libraries do not share their content with dandelon.com and just buy data generated by intelligentCAPTURE like Library of Congress, British Library, German National library.

Technically “dandelon.com” is based on IBM Lotus Domino server with the build in GTR search engine and intelligentCAPTURE is based on IBM Lotus Notes client. Each user query is analyzed for best match with thesauri
entries; e.g., three words typed in by the user may generate a query of up to 20 or 100 final search terms. It is running on one SUN server and replies to about 1 million clicks per month. “dandelon.com” is a service
of AGI – Information Management Consultants.

Similar applications are in use for searching for parliamental documents and for the website of a public administration of “Vorarlberg” in Austria.

SkiBonk the Resort, but Don’t Bonk Your Head

January 13th, 2009 by Rafi Farber
Posted in Verticals | 1 Comment »

Thinking about going skiing this winter? It happened once that I, a Miami native, gleefully decided to do such a thing one winter in college. Thinking I could snowboard because I used to skateboard, I got on one, took a short lesson, and proceeded to injure myself several times in the form of several bonks to the head that I ended up surviving, though not without suffering dain bramage.

But if you think you can stand up to the pummeling discipline that falling onto icy snow at high speeds can inflict on your fragile human body, think no further than SkiBonk. If you’ve got a place in mind – and it doesn’t even have to be an actual resort, it can be a general area – then SkiBonk will tell you all you need to know about the place. Directions, how much snow, what kind of snow, forecasts, videos about the particular resort, all in one package, all in one map in the middle of the screen.

You can search ‘city’, ’state’, ‘province’, ‘country’, ‘continent’ ’ski resort name’, so we’re not just limited to the US and Canada here, though I don’t think they have Antarctica cataloged yet.

Ski Bonk is a mashup of SnoCountry, OnTheSnow, Weather Underground, the National Weather Service. (Weather Underground? Isn’t that the anti-war group that bombed the pentagon in the 70’s? Well now it’s a website! Thank goodness things have gotten more towards the recreational an less towards the explosive and deadly side. At least in some countries.)

Users can take advantage of the dropdown box on the right hand of the screen to select a data set, then click a column header in the detail box to visually display that information on the map. Searching SkiBonk before skiing will let you know almost exactly what you’re getting into minus the actual experience, and provided you stay vertical, you should be fine. If not, stick to cross country, a sport I never really understood, and the winners of which at the winter Olympic games seemed very tired, cold, and crazed.

SkiBonk – search the snow, get your skis, don’t fall over, and have a great winter!

A Federated Search Primer – Part III of III

January 13th, 2009 by Guest Author
Posted in Federated Search, Guest Authors | 3 Comments »

Here is Part I and Part II

Federated Search: Beyond the Basics

We have only scratched the surface of the technology. I recommend playing with federated search applications and reading some of my other articles:

Some federated search applications include:

* Mednar – Searches medical information sources.
* Biznar – Searches business-related sources.
* WorldWideScience.org – Searches science content from all over the world, from government agencies, as well as other quality research and academic organizations.
* http://search.smartlib-bibliogen.ca/zengine?VDXaction=ZSearchSimple – Searches Capital SmartLibrary Consortium of Libraries.
* http://osulibrary.oregonstate.edu/metafind/about.html – Searches Oregon State University’s Library.
* http://scienceroll.polymeta.com/search/ui7/searchfr.jsp?un=scienceroll – Searches a medical student’s journey inside genetics and medicine through web 2.0.
* Science.gov – Searches science documents from a number of US federal government agencies.
* http://lifesearch.indexdata.dk/# – Searches University of Copenhagen’s Library of Faculty of Life Sciences.
* Scitopia.org – Searches digital libraries of leading science and technology societies.
* http://www.techxtra.ac.uk – Searches 31 different collections relevant to engineering, mathematics and computing, including content from over 50 publishers and providers.

Recommended reading:
* 99 Resources to Research & Mine the Invisible Web
* Beyond Google: The Invisible Web
* Crawling vs. deep web searching
* Glossary of search industry terms
* Introduction to the deep web
* Invisible Web: What it is, Why it exists, How to find it, and Its inherent ambiguity

deepweb
* Searching the Deep Web (video) Click here for the video.
* The trouble with general search engines
* What determines quality of search results?
* What is a connector?

Questions? Leave a comment for Sol.

solSol Lederman is the primary author of the Federated Search Blog, a blog sponsored by Deep Web Technologies and dedicated solely to the federated search industry. He also writes for the U.S. Department of Energy’s Office of Scientific and Technical Information (OSTI) Blog, primarily covering OSTI’s accomplishments and technologies. Sol’s first love is mathematics; he enjoys giving away prizes to people who can solve math problems that he presents through his personal blog, Wild About Math!.

You can read his series on Federated Search on AltSearchEngines here for Part I and here for Part II.

Many thanks Sol! -editor

New UK Job Search Engine JobDaddy

January 13th, 2009 by Charles S. Knight
Posted in Job Search, Newcomers, Verticals | 3 Comments »

jobdaddyJobdaddy is a job search engine. We collects jobs from job boards, agencies and employers across the UK. Our aim is to give job seekers a ‘one stop shop’ when it comes to searching for jobs online in UK. With Jobdaddy you can drill down to exactly what you’re after by refining by keyword, job title, location and industry etc.