Microsoft Sharepoint Products and Technologies Resource Kit eBook

596 Part VII: Information Management in SharePoint Products and Technologies
Word Lists
Shadow Indexes
Master Merge
When a document is indexed, the index information goes first to a word list. A word
list is a small, temporary index that contains index information for only a few documents.
Creation of a word list is very quick and does not require updating of any
stored data in the master index. This saves changes so that a merge with the master
index can take place less often.
Word lists are stored in memory and are not updated. When another set of documents
is indexed, another word list is created.
When the number of word lists exceeds the maximum allowable number, the Share-
PointPS Search service performs a shadow merge, which transforms word lists into
a shadow index. Shadow indexes are highly compressed and are saved to a disk,
which frees up memory that was occupied by word lists. The number of word lists
that accumulate before triggering a shadow merge can vary based on settings in
SharePoint Portal Server. See the “Balancing Resources” section of this chapter for
more information.
The master index is persistent, disk-resident, highly compressed, and optimized for
querying. Word lists and shadow indexes are part of the entire catalog that makes up
the content index, so queries from search requests are performed against all word
lists and shadow indexes as well as the master index. Having a large amount of the
catalog in the form of word lists uses a large amount of memory. Also, because queries
are performed against the master index as well as all shadow indexes and word
lists, a large amount of either or both of these can slow down search queries. By
default, the SharePointPS Search service performs a master merge whenever a full
crawl or propagation is completed or a threshold of a maximum number of word
lists and shadow indexes is reached. A master merge combines shadow indexes with
the master index.
Balancing Resources
A master merge is triggered when the number of word lists and shadow indexes
reaches a maximum threshold. The use of word lists uses memory resources while
saving on processor resources that would otherwise be used performing master
merges. However, because each word list must be searched during a query, too
many word lists can affect search performance. Similarly, shadow indexes can
merge word lists and avoid the resource-intensive process of master merges, but
again too many can slow search performance. The Balancing Resources setting in
the Central Administration site allows an administrator to balance the use of word
lists and shadow indexes with resources used by master merges and the performance
of search queries.