Microsoft Sharepoint Products and Technologies Resource Kit eBook

Plug-Ins
Chapter 21: The Architecture of the Gatherer 589
in a single document. If a document has more text than the limit, SharePoint Portal
Server stops the indexing process, considers the document indexed, and moves on
to the next document.
SharePoint Portal Server has a registry value that limits the maximum file size that
it will crawl. The registry value is called Max Download Size and is located in the
HKLM/Software/Microsoft/SPS Search/Gathering Manager key. By default, the entry
for this registry value is set at 16 MB. SharePoint Portal Server also has a registry entry
that defines how many times larger the file can be than the text. This registry value is
contained in the HKLM/Software/Microsoft/SPS Search/Gathering Manager/Max Grow
Factor key. By default, this value is set at four. This means that the text in a file can be
only up to four times larger than the file size.
After the protocol handler and IFilter have been loaded, the gatherer then connects
to the content source and begins streaming data out of the content source, obeying
the rules you’ve created for the content source. (Please refer to Chapter 22 and the
discussion on creating and managing content sources to learn about the Site Hit Frequency,
Site Path, and other rules that can be created as part of the content source.)
The gatherer first streams out the metadata and then streams out the content of
the document. The metadata includes the document’s properties and its permissions.
Permissions gathered from the document are referenced when the result set is
built, and any documents the user does not have access to are clipped from the
result set before it is presented back to the user.
The data streams are sent through a series of plug-ins, which are components
that perform certain functions on the data stream. By default, four plug-ins—
Schema, Indexer, PQS, and AutoCat—ship with SharePoint Portal Server 2003 and
are used to perform different functions.
Schema Plug-In
The Schema plug-in is a new feature in this version of Search for SharePoint Portal
Server. This plug-in is responsible for adding new objects to the schema used by
SharePoint Products and Technologies as they are discovered during the crawling
process. When needed, this plug-in is also responsible for alias name resolution to
Active Directory account names. This is useful for the profile database.
Indexer Plug-In
The Indexer is responsible for several functions, including word breaking, stemming,
and noise word removal. Each of these is discussed in the following sections.
Word Breakers
The data stream from the content source is an unbroken string of Unicode characters.
Word breakers are needed to determine where the word boundaries are so that the