Pump Up The Volume: An Assessment of Voice-Enabled Web Search on the iPhone:
a performance analysis of voice-enabled  mobile search services published today by MSearchGroove, found that ChaCha, the fastest growing SMS mobile search service in the industry, “proved superior” to two other voice-enabled search options for the iPhone: the Google Mobile App with Voice and Vlingo for iPhone,  a voice enabled application that allows users to direct their spoken queries to Google or Yahoo! (For the purposes of this study Vlingo provided a spoken interface to the Yahoo! search engine.)

ChaCha proved to offer exceptional results, with its human guides interpreting the search query accurately in the majority of cases. According to the study, ChaCha interpreted natural language search queries, that is, queries asked as questions, accurately in 94.4% of the tests and delivered an accurate search result in 88.9% of cases. The Google voice recognition technology interpreted queries accurately in 16.7% of tests and delivered accurate search results in 22.2% of tests. The Vlingo for iPhone voice recognition technology correctly interpreted queries in 72.2 % of cases and delivered accurate results (via Yahoo!) in 27.8% of tests.

“A clear finding that emerged from this study is that the use of human agents to help interpret spoken queries and conduct searches makes a positive difference in the quality of results delivered when compared to traditional search engines that use algorithmic software to find requested documents or information on the basis of keyword matches,” said Peggy Albright, the study’s lead analyst and principal author.

To evaluate the overall performance of voice-enabled mobile services offered by ChaCha, Google and Vlingo for iPhone with Yahoo!, the researchers asked a series of 18 queries representative of six typical mobile search categories (Navigational, Directions, Information Local, Information General, Social, and Long-Tail). For each query the researchers evaluated nine performance characteristics including response time, results accuracy, voice recognition accuracy, number of results received, keytaps required, relevancy of the result, location awareness, use of advertising, and presence of other value-added features. The study further took into account that a service could deliver its search results in the form of answers (as ChaCha offers) or as links to Web pages (which Google and Vlingo deliver); for each query tested, an accurate result could be achieved in either form.

The researchers also conducted two rounds of tests, using both natural language (speaking the questions in sentences) and keyword (reducing questions to a few specific terms) queries.  ChaCha, a natural language search engine, also outperformed both Google and Vlingo with Yahoo! in its ability to deliver accurate results in response to keyword queries, a traditional strength of universal search engines such as Google and Yahoo!

Both the Google Mobile App and Vlingo for iPhone use automated voice recognition technologies, while ChaCha leverages the use of actual human agents, or Guides, to interpret and transcribe spoken search queries.  The study shows automated voice recognition programs frequently misinterpret words – often the entire question gets lost in translation.

“What this research tells us is what we’ve known all along, if you want your consumers to hear your message, you’ve got to be able to hear them,” said Scott A. Jones, CEO of ChaCha. “Mobile search users are busy, on-the-go people, and they don’t have time to field incorrect answers due to voice recognition error; nor do fast-paced customers want to click-through links to get to their responses. With ChaCha Voice, you call our free number and ask your question – that’s it. Then a real person actually listens and transcribes your message, finds the correct answer and returns a concise SMS response in minutes.”

The report also identified a key advantage ChaCha has over its competitors: Its keen focus on social search, an approach that effectively infuses human preferences and human judgments into computer algorithms to pinpoint truly relevant information and potentially better answers.

“In mobile search, where algorithmic search can deliver neither personalized search results nor peer recommendations, social search and variations that tap the wisdom of crowds to deliver the right mix of answers and entertainment have a clear competitive edge over the plain-vanilla search we know from what Google and Yahoo! offer on the PC as well as mobile,” said Peggy Anne Salz, MSearchGroove founder, chief analyst, and co-author of the report. “Beyond tipping the scales back in favor of results that are relevant rather than search- engine optimized, social search also lays the groundwork for a conversation with people on their terms, paving the way for the delivery of mobile advertising that is relevant and more likely to be appreciated.”

Another factor playing in the favor of social search on mobile devices is the progress vendors have made to deliver people a good user experience. “Speech technology has steadily improved, and mobile phones provide an incentive for its use with their small size, safety considerations, and always-there microphone,” noted Bill Meisel, President, TMA Associates, and Editor, Speech Strategpy News. “This adds a dimension to mobile search that broadens the battlefield as companies fight for domination in this important sector.”

The complete report is available for free download from MSearchGroove.

About voice search

The use of voice-enabled technologies to provide access to mobile phone features and functions is an increasingly important area of innovation that offers new conveniences to customers and service differentiators and revenue opportunities for operators. The new study offers the industry a careful and comprehensive evaluation of how three services perform on one device—the iPhone 3G—which was the predominant device used for mobile Internet access and search during the study period.

About MSearchGroove

MSearchGroove is a leading source of in-depth analysis and commentary on mobile search, mobile advertising, and social media. Through primary research and exclusive C-Level interviews, MSearchGroove is the essential read for companies seeking ways to monetize their digital assets, drive mobile advertising revenues and harness the power of empowered consumers. Peggy Anne Salz, Founder and Chief Analyst, is an authority on mobile search and content discovery technologies. She has established a successful consulting career based on vision, insight, versatility, and over 15 years of industry experience. For more information, please visit www.msearchgroove.com

About Peggy Albright

Peggy Albright, owner of Albright Communications, is a veteran writer in the wireless industry, having covered the industry since 1996. She has contributed to leading U.S. and international publications including FierceBroadbandWireless and FierceWireless, the IEEE Computer Society, numerous Informa Telecoms & Media titles, MSearchGroove, Wireless Week, and others.  Through her business, Albright Communications, she provides research, writing, and editorial services to businesses and individuals in the wireless and high-tech sectors.

and if that doesn’t convince you to try ChaCha, maybe this video will: