Microsoft Sharepoint Products and Technologies Resource Kit eBook

590 Part VII: Information Management in SharePoint Products and Technologies
string can be broken back into individual words. SharePoint Portal Server uses word
breakers provided by Windows 2000 Server Indexing Service and also provides word
breakers of its own. The word breakers provided by the Microsoft Windows 2000
Server Indexing Service are for documents that are in Dutch, Italian, Swedish, and
German. SharePoint Portal Server provides word breakers for English, French, Spanish,
Japanese, Thai, Korean, Chinese-Traditional, and Chinese-Simplified.
If multiple languages are used in a single document, SharePoint Portal Server
recognizes that multiple word breakers are needed. If no word breaker is available
for a particular language, the neutral word breaker is used. In this case, words are
broken at neutral characters such as spaces and punctuation marks.
Stemmers
Many words in a language have several inflections that can be used. In English, even
a simple word such as “get” can also take the form of “getting,” “got,” or “gotten.”
Because of this, a component is needed to convert different variations of a word.
Components that convert all the variations of a word are called stemmers. Stemmers
also affect the formats of numbers, dates, and times so that they are handled consistently.
Stemmers are used only in processing queries and are not used in the indexing
process.
Noise Word Removal
Some words in a language are not useful for performing searches. Put another way,
these words provide no search value when executed in a search query. For example,
in the English language, words such as “the” and “a” are not useful in a search
because they provide no real search value: nearly every document will have these
words, making them indiscriminate in a search query. Words like these are considered
to be noise words. Noise words are different in each language, so, as with word
breakers, SharePoint Portal Server uses several different files to contain noise words.
Each file is a list of words that are removed during the indexing process.
Noise Word File Management
By default, SharePoint Portal Server stores noise word files in the \Program
Files\SharePoint Portal Server\DATA\Config directory of the server. The location of
the DATA directory can be changed during the server installation process.
Another set of the same noise word files is copied to the to local_drive\Program
Files\SharePoint Portal Server\Data\Applications\Application UID\Config directory.
These files can be used to specify noise word files that apply at the application
level instead of at the server or server-farm level. For example, if SharePoint
Portal Server and Microsoft SQL Server are installed on the same server, each
can have different noise word lists. Table 21-1 lists each language supported
and the corresponding noise word file included by default in SharePoint Portal
Server 2003.