I’m trying to find the quote regarding the cost of building an index of the internet, so I went to Google and ran “ballmer cost of building an index of the internet.” The #1 result is this page, so that’s funny. I’m thinking the quote is not from him. By the way, a search for “ballmer $300 million” produces a lot of interesting result, but not what I’m looking for.

Anyway, I need to get back to work, but for the moment I’m going to continue to believe that building and maintaining a comprehensive index of the internet is massively expensive, which is why very few companies are doing it.

Costs of indexing the entire Internet are much lower than reported. The $100M from SearchMe CEO is IMO completely off base, as a search professional and founder of two search startups (one vertical, one general) I can tell you that I have no idea where that figure is coming from. First, nowadays with multi-terabyte disks, you can fit billion+ index onto a single cluster within one rack. Also with wholesale bandwidth price dropping all the time (e.g. 1 Gbps of crawling bandwidth well under $10K), gigantic indices are withing the reach of many actually.

The problem is much more that doing such an index requires a custom, from the scratch effort as there is nothing in the marketplace off-the-shelf (incl open source) that can handle billion plus docs. The cost of resources and complexity in search comes much more from requirements of replicating clusters to handle massive query flows than putting up a single cluster (which is not easy in itself).

There are also other ways of amortizing costs, e.g. distributed, also other dark horses such as SSD which are only in its infancy wrt their impact on search.

Vertical search engines are very clear value propositions, by definition they are much more targeted and focused so they can achieve (much) higher CPMs. Sizewise, ther is no issue as there are many tools that can handle 10M+ vertical index that is typical in vertical search across pretty much all categories.

Regarding search volume, it is definitely true that there is (always) a fixed cost component. However, that cost is always going down too due to productivity improvements. SEO is the key factor here, if one can create a (vertical or general) index with millions of docs , largely indexed by Google, then you have a viable >$10M/yr business. To understand that , just take a look at Google results for any commercial query (e.g. ‘bmw m3′ or ‘canon g10′) and you will see how fierce the competition is.

On top of all that, it is natural for later entrants to push the envelope on marketing covering searches for lower and lower volumes since there is less or no competition there. Technology as well as human (outsourcing) elements are already in place.