Today we are very fortunate to have another blogger’s perspective on the post that we did on alternative search engines, asking which of the ”Alts” crawled the web and why:    “To crawl or not to crawl? The Alts speak out!” 

Александра (Alexandra) with SE la vie (get it?) sent this to us from beautiful Belarus!

She translated the quotes of different alternative search engines discussing the problem of crawling and indexing the web from my English post to Russian for her blog, and then reversed the process for this post!

Why is this problem acute? Probably mostly because “alternative search engine” is not a shaped definition yet (not speaking of just “search engines”). There are a lot of opinions. All of them are kind of right. And we all know this one: a search engine is a service which absolutely should have its own index and crawler. I already spoke my mind in my previous article “Alternatives for newbies:” “I will not set up any conditions like having its own index, algorithm or a crawler.”

And I still hold to that point. With all the variety of alternative search engines it’s not hard to find ones which initially, by their own idea, do not need an index (not speaking of meta-search engines).

But at the same time saying that a search engine is an index and a crawler – is quite correct. This is a principle of the majors to crawl and index everything on its way J But if there’s a vertical real estate search engine which should not crawl anything which is not related to real estate, then why should it care about crawling, say, a gadget blog? Just to be among those 11 alts which really crawl? That is why I would not be so fast to say that this or that search engine does not have a crawler so it is not a search engine at all. May be it does. You just don’t know about it. And may be you are not supposed to.
Read the rest of this entry »