By Guest Author Kathleen Dahlgren, PhD

This post is to add to the dialog about precision and recall which are standard measures of Search engine performance. Precision is a measure of retrieval accuracy calculated by dividing the total number of relevant retrievals by the number of all retrievals generated by the Search.  Recall is a measure of the extent to which relevant material in the total document base is found.  It is calculated by dividing the number of relevant retrievals by the total number of potentially relevant retrievals in the document base.

Pattern-matching technologies perform with both low precision and low recall (typically under 20% for both).   The TREC (Text Retrieval Conference), sponsored by the  National Institute of Standards and Technology (NIST), is a recognized source of precision/recall testing for various technologies, including pattern-matching and statistical approaches.  In TREC’s legal track competition in 2007, there were 13 technologies participating.  Their precision performance ranged from under 1% to 23% and their recall performance ranged from under 1% to 22%.

While Cognition did not participate in the TREC competition in 2007 (but is participating in 2008), it did conduct its own internal precision/recall tests on a wide variety of document bases (similar to the TREC data) and Websites.  These included the National Library of Medicine’s MEDLINE™, the public domain Enron fraud case, the public domain Microsoft anti-trust case, the BBC World News Website (http://news.bbc.co.uk/),  and the Global Issues Website (http://www.globalissues.com), among others.  For each test, 50 queries that were considered likely to be asked by users of the data/Website were formulated and posed to a CognitionSearch Search function on the sites’ documents.  Relevancy was judged for a sample of 20 or fewer retrievals and extrapolated.  Cognition’s precision exceeded 90%.  Recall was measured relatively.  In other words, full recall was taken to be the total of all relevant retrievals returned by any of the Search engines used in the particular test.  Cognition’s relative recall in these tests exceeded 90% relative recall.

We and other semantic technologies believe that by employing Semantic NLP technology, Search results will achieve significantly better precision and recall than pattern-matching or statistical approaches.