Spotlight on Spotlight - Carol Smith Home Page

Smith 16
Query #3
Search String Virginia or VA
Results Set
This search returned 5 documents, 4 of which were
relevant.
Precision 1 4/5, or 80% of all retrieved documents are relevant to the
query.
Recall
4/32, or 12.5% of all relevant documents were retrieved by
the query.
2_______
1 + 1
.125 .8
Harmonic
Mean
= 2_______
9.25
Observations
= 0.216
Recall was extremely low, because the search string was not
interpreted by the system as the user anticipated. <strong>Spotlight</strong>
does not recognize “or” as a valid Boolean operator. Those
documents that were retrieved happened to refer to the
state of Virginia as both ‘Virginia’ and ‘VA’, and included at
least one word containing ‘or’ as a letter sequence (i.e.,
“memorial”; “born”; “Dora”). Precision was high, but as
the search string was not interpreted as expected by the
user, it cannot be attributed to a well-formulated query.
1
<strong>Spotlight</strong> does not employ a ranking algorithm, and returns documents in lexicographic order by document
name. For this reason, precision cannot be presented at intermediate levels of recall
Additional observations:
As can be seen in the histograms on the following page, precision and recall display
an inverse relationship for all three queries.
Query 3 has the highest precision, but also the lowest recall, whereas query 2 has the
lowest precision, but the highest recall of the three queries.
Harmonic mean is poorest for query 3, reflecting the large difference between
precision and recall measures.
The data set is fairly small (144 documents), and this analysis may not properly
reflect the poor recall performance associated with searching large document
collections (Blair & Maron, 1985).