I am the Founder and CEO of Duck Duck Go, a new general purpose search engine. There have been a bunch of articles on our features, and I’ll mention some below, but I thought it might be fun to back up and talk about some of the motivation behind starting our search engine.

Yes, I was motivated by the unbelievably profitable search market. Who isn’t? But on the product side, I was motivated by two things primarily. First and foremost, Google et. al.’s result pages don’t seem to make sense to me, literally. They are made up of collections of automatically generated titles and snippets (often gamed through SEO). When taken together, these pages look like monolithic blobs of confusion to me, made even more confusing by similar looking advertisements. As a result, you find yourself clicking forward and back a lot, trying to figure out what’s what and if anything has the info you’re actually looking for.

At the same time, over the past decade, tons of great user generated content sites have been created, most notably Wikipedia. But there are 100s of others, e.g. Crunchbase, Wikia, LinkedIn, etc. These sites, by contrast, contain links to things and information about them that are edited and crowd-sourced by actual people. Consequently, their info is highly relevant and have titles and descriptions that actually make sense!

So I thought, look, what if we took all that great content and used it to improve search. And that is exactly what Duck Duck Go does. There are all the results you are used to *plus* all this good stuff from these human powered sources. Our rule is to not display anything from them unless we think the information is better than algorithmically generated results. That’s why we put the UGC stuff on top of the auto-generated stuff. Now we are taking the next step of transforming algorithmically generated results into things that look more like the other results. That is, changing the look and feel of them to make them more sensible.

The second motivating factor was spam. I increasingly noticed spam in Google et. al.’s results. Spam of all kinds. At the extreme are sites with absolutely no content and nothing but ads. In the middle are sites with little useful content and a lot of ads, so-called made for Adsense sites. Then there are sites that have content but are manipulating Google via SEO to appear under search terms for which they are not really relevant.

Duck Duck Go does a two-prong attack on spam. First from the bottom-up, we simply omit over 40 million domains that we believe are parked/spam domains. We partnered with The Parked Domains Project, which I am also involved in, to identify these domains. That project regularly crawls the Web looking for this stuff. I’m not sure why Google and others don’t do this as aggressively. Perhaps they are worried about too much censorship? Nevertheless, we’ve found a lot of these sites are in their indexes, but not in ours.

Then from the top-down, we use a lot of data from these user generated content (UGC) sites, which I mentioned above. Not only do they have great content, but they are also policed for spam by editors. So you find very little spam in them, and when it does occur, it is usually corrected quickly. Using both these approaches, Duck Duck Go has noticeably, and more importantly significantly, less spam.

Now, it turned out that by taking this approach of using UGC sites, we could introduce some really cool new features not seen in other search engines. For example, Duck Duck Go has special Category Pages and Meaning Pages. Category Pages group topics about similar concepts and Meaning Pages group topics with similar names. These special pages enable you to discover useful information about your search that aren’t in normal search results. For example, our White House Category Page links to the West Wing topic and its Meaning Page links to the White House, TN topic. That is, Duck Duck Go has ambiguous keyword detection! For example, try typing in the word apple.

Similarly, we could introduce real related topics, not just “related searches.” Our related topics are really related to the topic listed! All of this info, along with topic summaries, are displayed in a red “Zero-click Info” box, giving you information with zero-clicks. In effect, in addition to being a regular search engine, Duck Duck Go is also a “topic search engine,” displaying useful content about topics where appropriate. This squares with our primary goal to get you the information you’re looking for quicker and with less mental effort.

Gabriel Weinberg
CEO DuckDuckGo

This entry was posted on Wednesday, April 15th, 2009 at 2:00 pm and is filed under CEO Views. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.