Spotlight on Spotlight - Carol Smith Home Page

<strong>Spotlight</strong>
on <strong>Spotlight</strong>
An evaluation and review of Mac OS X
tiger’s desktop indexing application
Carol Smith
Info 624 – information retrieval systems
Summer 2005, Buzydlowski
Submitted august 17, 2005

Smith 2
TABLE OF CONTENTS
ABSTRACT 3
AUTHOR KEYWORDS 3
INTRODUCTION 3
Problem Domain and Scope 3
<strong>Spotlight</strong> Features – Brief Overview 4
DATA SET 6
Data Set Proposal 6
Data Set Description 6
Sample Document 7
Data Set Creation 7
Data Set Issues 8
EVALUATION 9
Methodology 9
1. Functional Analysis 10
2. System Performance Evaluation 11
3. Retrieval Performance 14
SYSTEM REVIEW 18
IR Model 18
Text operations 18
Text languages 18
Query language and operations 19
User interface, retrieval issues 19
CONCLUSION 20
BIBLIOGRAPHY 21

Smith 3
ABSTRACT
As the number of text and multimedia files stored by the average computer user continues to
increase, so will the need to effectively index and access one's 'personal digital library'. Apple
Computer, Inc.’s latest operating system, Mac OS X 10.4 (dubbed ‘Tiger’), includes an
integrated indexing and retrieval application known as ‘<strong>Spotlight</strong>’. This paper analyzes
<strong>Spotlight</strong>’s capabilities and limitations by first defining a data set of textual documents, and
then testing <strong>Spotlight</strong>’s performance in indexing and accessing the data set. The basic
features of <strong>Spotlight</strong> are introduced, and then the test data set is described, including issues
related to its creation and use. A three-pronged approach was adopted to assess the
performance of <strong>Spotlight</strong>. During the functional analysis phase, <strong>Spotlight</strong> was tested for
system errors. A performance analysis then assessed the speed and storage requirements of
the indexing system. Finally, a retrieval performance evaluation tested <strong>Spotlight</strong> against the
data set for precision, recall and harmonic mean measurements. The paper concludes with
observations about <strong>Spotlight</strong>’s underlying information retrieval model, in terms of text
languages and operations, query languages and operations, and interface issues. <strong>Spotlight</strong> is
found to be a utility with great promise, but also with significant challenges.
AUTHOR KEYWORDS
Apple; Mac OS X 10.4; Tiger; <strong>Spotlight</strong>; information retrieval; operating systems; evaluation.
INTRODUCTION
Problem Domain and Scope
As the number of text and multimedia files stored by the average computer user continues to
increase, so will the need to effectively index and access one's 'personal digital library'. Both
Microsoft and Apple Computer have recognized this growing need to manage digital
collections, and have been racing to integrate information retrieval utilities into their
competing operating systems. Microsoft’s Window Vista operating system (formerly known
as ‘Longhorn’) is slated for release sometime in 2006, and is expected to include integrated
indexing/query capabilities. Apple Computer, Inc. (hereafter, ‘Apple’), however, ‘beat them
to the punch’, releasing Mac OS X 10.4 (dubbed ‘Tiger’) to the public on April 29, 2005.
Included in Mac OS X Tiger is an integrated indexing and retrieval application known as
‘<strong>Spotlight</strong>’.
Mac users of greatly varying technical ability are already actively using <strong>Spotlight</strong> to seek and
retrieve information from their desktop computing environments; indeed along with their
favorite Internet browser and search engine, it is likely to become the information retrieval
system they access most often. As a developer's tool, the <strong>Spotlight</strong> search engine will also be
incorporated into dozens of third-party applications. Given its potential for wide use, a
thorough analysis of <strong>Spotlight</strong>’s information retrieval capabilities and limitations is
warranted. Published reviews of <strong>Spotlight</strong> are glowing with praise, but unfortunately provide
little analytical detail. This paper seeks to fill that void by systematically evaluating <strong>Spotlight</strong>’s

Smith 4
performance. To accomplish this, a data set of textual documents is first created and defined,
and then used to test <strong>Spotlight</strong>’s performance in indexing and accessing the data set.
<strong>Spotlight</strong> Features – Brief Overview
<strong>Spotlight</strong> indexes the contents of a drive automatically, with no explicit action required by
the user. For textual documents, the full text is indexed, and for all file types, application
metadata is also indexed. The <strong>Spotlight</strong> engine works closely with the Mac operating system,
updating the index anytime a new file is created or modified. Additional detail about
<strong>Spotlight</strong>’s indexing architecture and processes is provided in the Indexing Processes and
System Review sections of this paper.
Because the <strong>Spotlight</strong> indexing/retrieval engine is an integrated component of the Mac
operating system, it is ‘always on’ and doesn’t need to be launched in the manner of a
traditional application. The <strong>Spotlight</strong> query window is always available within a few
keystrokes, and can be accessed in a number of alternate ways:
1. The upper right-hand window of the Mac interface contains a permanent ‘spyglass’
icon that is always within view– clicking on this icon brings up the basic <strong>Spotlight</strong>
query window:
2. The same query window can alternately be reached via a command-space bar
keystroke combination.
3. <strong>Spotlight</strong> query windows are built into popular Apple applications and utilities,
including Mail, Preferences, Address Book, Calendar and others.
4. <strong>Spotlight</strong> queries can also be executed within the Apple Finder window. This final
access method also permits the creation and saving of customized queries called
‘Smart Folders.’ Apple envisions these virtual folders as a new method for organizing
and managing information, one that may at least partially supplant traditional
physical file organization.

Smith 5
As keywords are entered into the <strong>Spotlight</strong> query window, matching documents on the drive
are listed, ordered first by general file type, then lexicographically by file name. Clicking on
any file name will open the document within its associated application:
A user can also hit the Return key, in order to open the results set in a separate window. This
window permits additional manipulation of <strong>Spotlight</strong> query results, including alternate
ordering options and additional filtering functions:

Smith 6
DATA SET
Data Set Proposal
In keeping with <strong>Spotlight</strong>’s anticipated use as an indexing and retrieval system for individual
digital collections, a genealogical data set of personal interest and utility to the author was
envisioned.
Genealogists spend a significant amount of time reviewing Internet message boards,
particularly those dedicated to family surname research. These message board services offer
sophisticated query capabilities, including features such as field searching, Soundex
searching, and imposition of date range limits. Despite such useful information retrieval
features, however, searching the message boards is still a time-consuming affair. Because of
historical name spelling variants, a message of interest might be posted on any of multiple
surname message boards. When researching the Minnick family, for example, a query must
be individually executed on as many as 12 different message boards (Minnick; Minick;
Minck; etc…) located on multiple servers, in order to conduct a comprehensive search.
There is currently no way to query multiple Internet message boards simultaneously, even
within a single web site.
By creating a unified data set of postings from multiple Internet message boards, a
genealogist should be able to utilize <strong>Spotlight</strong> to rapidly execute comprehensive searches via
a single query. Creation of the initial data set will requires a significant investment of time up
front, but should be rewarded by faster search and retrieval for subsequent information
needs. The created data set, in combination with <strong>Spotlight</strong>, will essentially enable ‘metasearch’
capabilities across multiple message boards.
Data Set Description
The test data set was drawn from six separate surname message boards, all hosted by
Ancestry.com (http://ancestry.com/share/):
Minnick Surname Board
Minick Surname Board
Mink Surname Board
Minnich Surname Board
Minich Surname Board
Minck Surname Board
Additional message boards are located at http://genforum.genealogy.com and numerous
other genealogical web sites; these would be included in any fully implemented project, but
were deemed nonessential for the paper’s primary purpose of evaluating <strong>Spotlight</strong>’s
indexing/retrieval performance.
To keep the project manageable, the data set was limited to discussion threads with an initial
posting dated 1/1/2004 or later. Any messages dated 1/1/2004 or later but related to a

Smith 7
thread initiated prior to 2004 were not considered for inclusion. These date limits resulted in
a data set of 144 plain text files, as well as 22 JPEG images that were attached to the original
messages.
The data set displays several interesting characteristics that may present information retrieval
challenges, including:
A high incidence of recurring terms (first names, dates, etc.)
Numerous variant expressions, including abbreviations and unintentional misspellings
(e.g., Mississippi; Miss.; Missisippi; MS)
Polysemy; that is, words with multiple possible meanings (e.g., Virginia as a place;
Virginia as a female name)
Sample Document
The below image is representative of a typical message board posting, in its original HTML
formatting.
Boards > Surnames > Minck
URL: http://boards.ancestry.com/mbexec/message/an/surnames.minck/9
Data Set Creation
Creation of the data set was a predictably tedious, manual affair. For each of the 144
individual message board postings, the following steps were followed, in sequence:

Smith 8
1. The target html page was opened within a web browser.
2. Because a straight copy/paste routine would have captured undesirable information
and hyperlinks related extraneous to the message, each page was then reloaded via
the page’s ‘Printer-friendly’ hyperlink.
3. The message was copied in its entirety using cmd-a/cmd-c keyboard shortcuts
(Macintosh).
4. Using the cmd-v keyboard shortcut (Macintosh), the message was then pasted into a
new plain text document, using Apple’s TextEdit application.
5. Two corrections were made to each plain text document:
a. The phrase “Return to Message” (a hyperlink in the original page) was
deleted from the end of each document.
b. In order to avoid web crawler agents, the original html pages provide e-mail
addresses in .gif format. For this reason, each author’s e-mail address
information needed to be entered manually.
6. Each plain text file was then saved to the hard drive.
7. A small percentage of message board postings were accompanied by .jpg
attachments, typically scanned documents relating to the message. Each of these
attachments (22 in all) was saved as separate data set files. Each attachment had to
first be loaded into a separate browser window, for some unknown reason,
attachments could only be saved as .gif images without this extra step, even though
the extension of the attachment indicated it was a .jpg file.
After some consideration, it was decided to name each text file sequentially, beginning with
001, 002, 003, etc. If an initial message board posting received replies, each posting of a
single thread were given the same number, but distinguished with sequential letters; e.g.,
001a, 001b, 001c, etc… Some thought was given as to whether file names should indicate
the level of depth in a particular thread; that is, if a posting was the second reply to a reply of
an initial posting, label it 001aab. This level of complexity was deemed unnecessary,
however, as any thread in question could be easily located in its original web location, should
the sequence of postings become of interest.
Data Set Issues
As described in the Functional Analysis section below, two decisions made during the
creation of the initial data set proved problematic, and required further data set modification:
1. Because Mac files do not require extensions (.txt, .doc, etc.), extensions were not
initially entered during the file-naming step.
2. Documents were initially saved to separate sub-folders for each of the Internet
message boards (i.e., “Ancestry-Minnick”; “Ancestry-Minick”; “Ancestry-Minck”;
“Ancestry-Minnich”; “Ancestry-Minich”; “Ancestry-Mink”). Attachments were
further segregated into folders within these folders, labeled “Ancestry-Minnick-
Images”, etc. Finally, all six subfolders were contained within a single top-level folder
labeled “<strong>Spotlight</strong> Data Set.”
It should also be noted that the fielded format of the documents in their original web format

Smith 9
was an initial attraction, as it offered up the possibility of testing <strong>Spotlight</strong>’s performance on
structural (syntactic) queries, as well as on semantic content. The fielded structure of data is
not retained, however, when information is converted to plain text format. This loss of
structural integrity was not anticipated (author oversight), but is hopefully made up for by an
extended <strong>Spotlight</strong> system review.
EVALUATION
Methodology
Evaluations were conducted by two independent users (hereafter, “User 1” and “User 2”).
One user was familiar with the contents of the data set, the other not. Neither user had prior
experience with the <strong>Spotlight</strong> interface, although they were both experienced users of
Macintosh operating systems. All tasks were conducted in batch mode (vs. interactive) mode;
that is, queries were executed and responses evaluated on an individual basis, rather than as
iterative sequences of query/response/revised query. Evaluations were carried out in a home
setting, but the structured nature of the tasks rendered the experiments closer to a laboratory
session than a real-life field assessment. All evaluations were conducted on a 1.07GHz Apple
iBook G4 laptop computer with 512 MB of DDR SDRAM.
A three-pronged approach was adopted to assess the performance of <strong>Spotlight</strong>:
1. Functional Analysis: During this errors analysis phase, users were asked to freely
utilize <strong>Spotlight</strong> to access the data set. Defined retrieval tasks were not provided;
instead, users created a range of their own ad hoc tasks intended to reveal functional
problems or inconsistencies in the system’s indexing and retrieval performance.
Users were asked to determine whether their chosen tasks were properly supported
by the system, and whether they were ultimately able to accomplish the tasks. Users
were asked to verbalize their experiences and challenges as these tasks were executed.
2. System Performance Evaluation: As a proprietary system (and due to the author’s
lack of technical prowess), obtaining precise calculations of <strong>Spotlight</strong>’s response time
and storage requirements proved challenging. Response time tests were executed
using a manually controlled stopwatch; calculations should therefore be only
considered as estimates. Further, an academic discussion of <strong>Spotlight</strong>’s storage
requirements was substituted for an actual evaluation, as no means for calculating
storage use could be determined by the author.
3. Retrieval Performance Evaluation: To assess the indexing and retrieval
performance of <strong>Spotlight</strong>, classic precision and recall measurements were calculated
separately. Harmonic mean, a measurement unifying precision and recall levels into a
single metric, is also provided.
Execution and outcome of these three evaluation modes are discussed in separate sections,
below.

Smith 10
1. Functional Analysis
Users were asked to first browse the data set and examine individual documents, in order to
devise a broad range of retrieval tasks. These tasks were then executed freely using <strong>Spotlight</strong>,
in an effort to reveal functional challenges. Performance errors became apparent at a very
early stage:
Task #1 (User 1): Locate all documents authored by Verna Williams.
Error: During the browsing session, User 1 pre-determined that at least 5
individual documents existed in the data set with author listed as either
‘Verna’ or ‘Verna Williams’. When entering ‘Verna’, ‘Author: Verna’ or
‘Author: Verna Williams’ in the <strong>Spotlight</strong>, however, zero results were
retrieved. Similar retrieval failures were consistently experienced with other
devised user tasks.
Problem Source: Several hours of exploration and experimentation revealed
the reason for this performance failure. All 144 plain text files in the data set
were saved without the .txt extension added to the document name. Such file
extensions are necessary in a Windows environment, but are optional in Mac
operating systems. In order for <strong>Spotlight</strong> to index a plain text file, however,
the .txt file extension is apparently required. Although not necessarily a
functional error, it is one that conflicts with long-established system
behavior, and will inevitably cause confusion for Mac users.
Resolution: As soon as the extension was added to all text files in the data
set, User 1 was able to successfully execute all remaining functional retrieval
tasks.
Task #2 (User 2): Locate all documents mentioning the name “John”.
Error: During the browsing session, User 2 observed that John was a
common name listed in the message board postings, and was curious about
what percentage of documents contained this name. Even after the .txt
extension had been added to all plain text files in the document set, however,
User 2 was unable to execute the retrieval task, receiving a results set of zero
documents.
Problem Source: Again, exploration and experimentation led to an
understanding of the performance failure. Whereas User 1 was utilizing
<strong>Spotlight</strong> to conduct a system-wide search of the computer’s entire hard
drive, and then analyzing just those .txt files with the anticipated file name
formatting (001, 002, etc.), User 2 chose to access <strong>Spotlight</strong> via the system’s
Finder feature, specifying a focused search of just the Data Set folder. As
previously described, this folder contains six subfolders, and documents were
saved within those six subfolders. It was determined that when <strong>Spotlight</strong> is
directed to examine the contents of a particular folder, it analyzes folder

Smith 11
contents only one level deep; that is, <strong>Spotlight</strong> was seeking keyword matches
on the folders themselves, but not on their contents.
Resolution: Although this may not be a functional error, it is regarded as a
serious limitation in <strong>Spotlight</strong>’s performance, and one that is likely to confuse
many users, whose prior computing experience will lead them to expect all
nested contents of a folder to be considered during a targeted search query.
To overcome the immediate problem, however, the situation was resolved by
removing all nested folder structures within the Data Set Folder. Once
accomplished, User 2 was able to conduct all remaining functional retrieval
tasks (see Appendix A) without error.
After this, no further functional errors were identified, and the functional evaluation was
concluded.
2. System Performance Evaluation
Performance evaluations assess the efficiency of a retrieval system’s architecture, in terms of
its use of storage space, system interactions and response time. Unfortunately, my technical
abilities are limited, and Apple provides no built-in utilities for assessing <strong>Spotlight</strong>
performance; after some research I was unable to identify any feasible methods for
generating precision performance metrics. In lieu of this, I elected instead to make rough
response time observations (using a manually controlled stopwatch), and to discuss
<strong>Spotlight</strong>’s indexing and retrieval structures in general terms.
2a. Response time: Because <strong>Spotlight</strong> is tightly integrated with the Mac operating
system, indexing of an entire file system is conducted as soon as a drive has been
introduced, and automatically updated each time a new document is created or
modified. This up-to-date system level index is readily available at any time for
searching by the user, via an icon in the upper-right hand corner of the interface. As
one begins typing a word, matching documents immediately begin filling the results
window, arranged by media type. As the word continues to be typed, non-matching
results are rapidly eliminated from the results window. Once the user has finished
typing the search query, the final results screen takes anywhere from 3-5 seconds to
stabilize. From a user perspective, then, initial response times appear nearly
immediate, with final results usually available within a 5-second time frame.
With ad hoc experimentation, two general response time challenges were observed
(neither specifically associated with the defined data set):
If a user executes a search query, and then quickly selects a particular file to
be opened while the results set is still ‘stabilizing’, system response time to
locate and open the chosen file slows slightly, from roughly 2-4 seconds to 4-
6 seconds elapsed time. It can also be difficult to select a file accurately while
the results set stabilizes, because file locations are continually shifting within

Smith 12
the results list. This, however, can be considered a user interface issue, rather
than a system performance issue.
When first connected to an external drive, <strong>Spotlight</strong> instantly begins indexing
the drive’s contents. A user can conduct search queries while indexing is
taking place, but response times are significantly slowed down, to between 5-
8 seconds. Additionally, one cannot consider the results set to be complete
until indexing of the underlying file system is finished. As an experiment, a
250GB external drive containing 38GB of data was connected to the primary
evaluation computer. <strong>Spotlight</strong> began indexing the drive at 11:35:40AM, and
finished at 12:33:11PM, an elapsed time of 57 minutes, 31 seconds (an
average indexing speed of 1 minute, 31 seconds per GB). During this time,
all executed <strong>Spotlight</strong> queries – even those executed on the primary
computer’s hard drive – were visible slowed by an additional 0-5 seconds per
query.
<strong>Spotlight</strong> results set – response time slows markedly
during the indexing of newly introduced drives.

Smith 13
2b. Indexing architecture: The <strong>Spotlight</strong> indexing process is nicely illustrated by
an Apple graphic (Apple Computer, 2005c, p.11):
Whenever a document is created or modified, or whenever a new drive is introduced,
<strong>Spotlight</strong>’s search engine initiates a query of the underlying file system, to determine
the type of file(s) involved. Once ascertained, <strong>Spotlight</strong> then calls upon the
appropriate plug-in to import content and metadata information. Every type of
file has an associated plug-in; many plug-ins come built-in with the Tiger operating
system, others can be created by developers to allow <strong>Spotlight</strong> indexing of less
common file types. After parsing a file’s contents, the information is populated in the
metadata index and the content index, as appropriate. Collectively, these two
indices are referred to as the ‘Apple Store’, and each drive maintains separate stores.
Users can then use the <strong>Spotlight</strong> search interface (or an independently developed
<strong>Spotlight</strong> API) to search the Apple Store, and connect to the appropriate
application, when a file of interest is located.
<strong>Spotlight</strong>’s indexing and information retrieval functions are tightly integrated with
the Mac operating system. Both Apple literature and third-party reviews tout this as a
significant advantage over add-on search tools, such as X1 for Windows. Certainly,
<strong>Spotlight</strong>’s automated indexing and always-available search field are convenient
features for users. Without a direct performance comparison against third-party
products, however, it’s difficult to assess the merit of such praise.
<strong>Spotlight</strong>’s content index is generated using Apple’s proprietary Search Kit
technology. The following statements in Apple’s developer literature (2004) describe
the use of an inverted file mechanism in Search Kit, with addressing granularity at
the document level only:
Inverted file mechanism: “A Search Kit inverted index lists each
constituent term exactly once, no matter how many of its contained
documents include the term and no matter how frequently the term appears
in any of the documents. In other words, the index tracks which documents
use the term, and how often, but the term appears in the index just once.”
Addressing granularity at the document level: “To Search Kit, a
document is atomic in that it defines the granularity of a search. Using Search

Smith 14
Kit, your application can find documents—as your application understands
them—but cannot locate the position of a term within a document.”
Although two other indexing methods are available to Search Kit developers (a
“vector index” and an “inverted vector index”), <strong>Spotlight</strong> is most likely using the
inverted index option, for the following reasons:
Apple (2004) characterizes the inverted index structure as being “faster and
smaller” than the two other Search Kit indexing methods, features essential
to <strong>Spotlight</strong>.
<strong>Spotlight</strong> can identify matching files, but cannot identify the location within a
file where specified text appears.
<strong>Spotlight</strong> provides keyword-based searching (as opposed to similarity
searching). Baeza-Yates, & Ribeiro-Neto cites inverted file mechanisms as
“currently the best choice for most [keyword-based search] applications”
(1999, p.191), and indeed, Apple (2004) recommends the Search Kit inverted
index option as the best option for keyword-based systems.
3. Retrieval Performance
Precision and recall are the two classic measurements of a system’s information retrieval
performance. Recall measures a system’s ability to retrieve all (known) relevant documents,
while precision measures the percentage of relevant documents in a particular results set. A
third measure, the harmonic mean, combines precision and recall measurements into a single
performance metric.
Although an inevitably subjective process, neither precision nor recall can be tested without
first identifying the subset of relevant documents for a particular information need. For this
reason, a retrieval task was designed in advance, and all pertinent documents within the 144-
document collection were identified. User 1 assessed document relevancy, as she was
familiar with the data set, and the task was then presented to User 2, who possessed no prior
contact with the data set. User 2 conducted 3 search queries, with the following results:
Relevant
Documents
Task 1: Locate all documents mentioning the state of Virginia
Of the 144 text documents in the full data set, 32 are deemed pertinent to the
information need:
005; 006; 007; 008; 009; 010; 011; 012; 013; 014; 015; 016; 017; 018; 020a;
020b; 020c; 022; 028a; 029; 030; 031; 032; 050; 051; 052; 063a; 065; 072d; 078;
080k; 081;

Smith 15
Query #1
Search String Virginia
Results Set
This query returned 17 documents, 13 of which were
relevant.
Precision 1 13/17, or 76.47% of all retrieved documents are relevant to
the query. 1
Recall
17/32, or 53.13% of all relevant documents were retrieved
by the query.
2_______
1 + 1
.5313 .7647
Harmonic
Mean
= 2_______
3.19
Observations
= 0.627
All 4 non-relevant documents were captured because the
author’s first name was ‘Virginia’, a polysemic issue.
Precision is high, but almost half of all relevant documents
were not returned – these all referred to the state of
Virginia as ‘VA” (or ‘Va’, in one case).
Query #2
Search String VA
Results Set
This search returned 37 documents, 25 of which were
relevant.
Precision 1 25/37, or 67.56% of all retrieved documents are relevant to
the query.
Recall
25/32, or 78.13%% of all relevant documents were
retrieved by the query.
2_______
1 + 1
.7813 .6756
Harmonic
Mean
= 2_______
2.76
Observations
= 0.725
Recall was quite high, primarily because the majority of
relevant documents referred to the state as ‘VA’ in the
subject line. A single author posted the majority of the
messages. A larger data set, reflecting the posting styles of
many different people, may not have yielded results as
favorable.

Smith 16
Query #3
Search String Virginia or VA
Results Set
This search returned 5 documents, 4 of which were
relevant.
Precision 1 4/5, or 80% of all retrieved documents are relevant to the
query.
Recall
4/32, or 12.5% of all relevant documents were retrieved by
the query.
2_______
1 + 1
.125 .8
Harmonic
Mean
= 2_______
9.25
Observations
= 0.216
Recall was extremely low, because the search string was not
interpreted by the system as the user anticipated. <strong>Spotlight</strong>
does not recognize “or” as a valid Boolean operator. Those
documents that were retrieved happened to refer to the
state of Virginia as both ‘Virginia’ and ‘VA’, and included at
least one word containing ‘or’ as a letter sequence (i.e.,
“memorial”; “born”; “Dora”). Precision was high, but as
the search string was not interpreted as expected by the
user, it cannot be attributed to a well-formulated query.
1
<strong>Spotlight</strong> does not employ a ranking algorithm, and returns documents in lexicographic order by document
name. For this reason, precision cannot be presented at intermediate levels of recall
Additional observations:
As can be seen in the histograms on the following page, precision and recall display
an inverse relationship for all three queries.
Query 3 has the highest precision, but also the lowest recall, whereas query 2 has the
lowest precision, but the highest recall of the three queries.
Harmonic mean is poorest for query 3, reflecting the large difference between
precision and recall measures.
The data set is fairly small (144 documents), and this analysis may not properly
reflect the poor recall performance associated with searching large document
collections (Blair & Maron, 1985).

Smith 17

Smith 18
SYSTEM REVIEW
Apple’s <strong>Spotlight</strong> application is tightly integrated with the Mac operating system. As such, its
indexing and retrieval operations are closely guarded proprietary processes, discussed only in
the broadest terms in corporate literature. Despite these restrictions, the above performance
evaluation provides a good understanding of <strong>Spotlight</strong>’s underlying information retrieval
model. What follows is a broad review of <strong>Spotlight</strong>’s information retrieval features, as
deduced from the evaluation process. Search and retrieval issues that became apparent
during the course of the evaluation are also noted.
IR Model
<strong>Spotlight</strong>’s underlying information retrieval model appears to be a classic Boolean model
with binary weighting; that is, a document is either relevant (included in results set) or nonrelevant
(excluded from results set). This is evidenced by the fact that results are presented in
lexicographic order by file name, with no ranking algorithm used.
Text operations
<strong>Spotlight</strong>’s logical view of documents is full-text, and includes information relating to
syntactic structure. Experimentation does not suggest the existence of any text normalization
procedures. Specifically, there appears to be:
No lexical analysis. For example, a <strong>Spotlight</strong> search on “full-text” will locate this
paper; but it is excluded from the results list if the hyphen is excluded in the search
query.
No stopword removal. A search of the word ‘the’, for example yields 12,040 matches
on the tested hard drive. This figure includes 1,1155 rich and plain text documents,
and 443 PDF documents.
No stemming. A search on the word ‘enter’ will return this paper in the results set,
but fails to do so if the same word is searched with an added suffix of ‘-s’ or ‘-ing’.
Text languages
<strong>Spotlight</strong> indexes a broad range of text languages, including both plain and rich text formats,
PDF documents, markup languages, and metadata for many common file formats.
Additionally, <strong>Spotlight</strong> can index multimedia files, system fonts and scripts, and applicationspecific
text such as e-mail messages, address book entries, etc. Additionally, <strong>Spotlight</strong> plugins
permit developers to expand <strong>Spotlight</strong>’s indexing coverage to handle less common or
newly developed text languages and file formats.

Smith 19
Query language and operations
<strong>Spotlight</strong>’s query functions are responsible for many of the system’s limitations. It permits
basic single-word or multiple-word queries on both text content and syntax (metadata), and
its internal division of words into letters allows matching on partial words.
Without an understanding of Unix command-line operations, however, users are unable to
execute even the simplest of Boolean queries, and cannot specify context queries such as
phrase or proximity searches. Apple either considers such querying to be beyond the
understanding of most users, or plans to release expanded search functionality in subsequent
operating system releases. Regardless of the reason, sophisticated querying is currently only
available to ‘power users’.
These query limitations are particularly troublesome because:
1. They are not transparent to the user. Internet surfers who use basic Boolean
operations such as AND, OR, and “phrase queries” may attempt to execute Boolean
operations, and fail to correctly evaluate the results set, as demonstrated by User 2
during the retrieval evaluation (see Query 3). The <strong>Spotlight</strong> interface provides the
user with no visual affordances as to proper (or improper) query formulation.
2. Language is a rich communication medium, offering a seemingly boundless diversity
of ways in which concepts can be expressed. As a full-text indexing system with no
apparent text normalization, <strong>Spotlight</strong>’s information retrieval performance is
particularly susceptible to the problems of synonymy and polysemy, and thus
particularly in need of sophisticated query operations. The “Virginia” retrieval task
(Query 1) used in this paper’s retrieval performance evaluation demonstrates this
aptly.
User interface, retrieval issues
<strong>Spotlight</strong> offers a ‘Smart Folder’ feature, permitting structural queries via a series of
dropdown menu choices that can be saved for future use. This interface was found to be
stilted, limiting in options, and failing in its support of essential query operations such as
phrase searching, Boolean OR statements, etc.
Smart Folders are envisioned as a means for users to access their files without regard to their
physical storage location. Once a Smart Folder is defined and saved, it updates itself
automatically, providing the user with an up-to-date list of all files meeting their specified
criteria. Instead of browsing multiple times through layers of nested folders, users can
potentially access all relevant information via a single, ‘virtual folder’. The Wall Street Journal
rightly points out the potential for this model to change users’ primary mode of information
retrieval:
“This is a big deal…<strong>Spotlight</strong> could spark a major change in the way people use
computers. Instead of hunting for documents or clicking on programs, people may
now start activities by searching for relevant files and then opening them as needed”
(Mossberg, 2005).

Smith 20
In its current form, however, the Smart Folder feature has one significant drawback, as
revealed during the system performance evaluation. Users will still sometimes want to focus
their query on a single folder. The Smart Folder feature permits this; unfortunately, the
search will be executed only one layer deep within the specified folder; the contents of any
nested folders are not considered by the query.
<strong>Spotlight</strong>’s Smart Folder feature.
Again, only experience and experimentation will reveal this limitation to the user; the
interface provides no guidance, no manual is provided, and Apple’s Help application is silent
on the issue.
CONCLUSION
Multiple published reviews of the new Mac OS X 10.4 operating system (‘Tiger’) were
consulted for this system evaluation. These reviews are uniformly positive in their
assessment of <strong>Spotlight</strong>, and so the project was approached with considerable optimism.
Having encountered many serious issues with <strong>Spotlight</strong>’s retrieval model, I am now
somewhat apt to believe these reviews were primarily derived from promotional materials
supplied by Apple, and involved scant independent analysis.
Put succinctly, the <strong>Spotlight</strong> indexing architecture is impressive; the indexing process is
transparent, automatic, and tightly integrated with the operating system. Without the
companionship of an effective information retrieval model and search interface, however,
the full power and utility of the index remains inaccessible to the average user.

Smith 21
BIBLIOGRAPHY
Apple Computer, Inc. (2005a). <strong>Spotlight</strong>. Find anything, anywhere, fast. Retrieved August 2,
2005 from http://www.apple.com/macosx/features/spotlight/.
Apple Computer, Inc. (2005b). Tiger developer overview series: Working with <strong>Spotlight</strong>.
Retrieved August 2, 2005 from http://developer.apple.com/macosx/spotlight.html.
Apple Computer, Inc. (2005c). Technology brief: Mac OS X <strong>Spotlight</strong>. Find anything on
your Mac instantly. Retrieved August 2, 2005 from
http://images.apple.com/macosx/pdf/MacOSX_<strong>Spotlight</strong>_TB.pdf.
Apple Computer, Inc. (2004). Developer Connection. How Search Kit Works. Retrieved
August 8, 2005 from
http://developer.apple.com/documentation/UserExperience/Conceptual/SearchKi
tConcepts/searchKit_concepts/chapter_3_section_5.html
Baeza-Yates, R., and Ribeiro-Neto, B. (1999). Modern information retrieval. New York:
ACM Press.
Beagrie, N. (June, 2005). Plenty of room at the bottom Personal digital libraries and
collections. D-Lib Magazine, 11(6). Viewed August 5, 2006 at
http://www.dlib.org/dlib/june05/beagrie/06beagrie.html.
Blair, D.C., and Maron, M.E. (March, 1985). An evaluation of retrieval effectiveness for a
full-text document-retrieval system. Communications of the ACM, (28)3: 289-299.
Coffee, P. (May 30, 2005). ‘Tiger’ invites developers in. eWeek, 22(22): 46.
Lewis, P. (May 16, 2005). Tiger tale: Look before you leap. Fortune, 151(10): 200, 202.
McElhearn, K. (August, 2005). Command <strong>Spotlight</strong>. Macworld, 22(8): 88-89.
Michaels, M. (September, 2004). 10 things to know about Tiger. Macworld, 21(9): 50-55.
Mossberg, W.S. (April 28, 2005). Tiger leaps out in front; Apple operating system offers new
approach to searching, Smart Folders, better browser. Wall Street Journal (Eastern
Edition), p. B1. Retrieved August 2, 2005 from ProQuest database.
Pogue, D. (April 28, 2005). Apple’s Tiger may even have PC owners longing for a Mac to
put it in. The New York Times, pp. C1, C10. Retrieved August 2, 2005 from Lexis
Nexis Academic database.
Wildstrom, S.H. (May 9, 2005). Tiger makes Mac’s edge even sharper. Business Week, 3932:
28.