MARKLOGIC SERVER

The fast diacritic sensitive searches option works the same way, but for diacritic sensitivity.
(Diacritics are ancillary glyphs added to a letter, like an umlaut or accent mark.) By
default, if you search for "resume," you'll see matches for "résumé" as well. That's
usually appropriate, but not in every case. By turning on the diacritic-sensitive index,
you tell MarkLogic to maintain both diacritic-insensitive and diacritic-sensitive term list
entries, so you can resolve diacritic-sensitive matches out of indexes. 3
STEMMED INDEXES
Stemming is another situation where MarkLogic provides optimized indexing options.
Stemming is the process of reducing inflected (or sometimes derived) words to their
root form. It allows a query for "run" to additionally match "runs," "running," and
"ran" because they all have the same stem root of "run." Performing a stemmed
search increases recall by expanding the set of results that can come back. Often that's
desirable, but sometimes not, such as when you're searching for proper nouns or precise
metadata tags.
MarkLogic gives you the choice of whether to maintain stemmed or unstemmed
indexes, or both. These indexes act somewhat like master index settings in that they
impact all of the other text indexes. For example, the fast phrase searches option looks at
the master index settings to determine if phrases should be indexed as word pairs or
stem pairs, or both.
In versions 8 and earlier, MarkLogic by default enables the stemmed searches option and
leaves word searches disabled. Usually that's fine, but it can lead to surprises—e.g., if
you're searching for a term in a metadata field where you don't want stemming. For
such cases, you can enable the word searches index and pass "unstemmed" as an option
while constructing the search query constraint. In version 9, the settings are reversed,
and MarkLogic by default disables stemmed searches and enables word searches.
MarkLogic uses a language-specific stemming library to identify the stem (or sometimes
stems) for each word. It has to be language-specific because words like "chat" have
different meanings in English and French and thus the roots are different. It's also
possible to create language-specific custom dictionaries to modify the stemming
behavior. For details, see the Search Developer's Guide. 4
3 MarkLogic follows the Unicode definition of what is and isn't a diacritic, which can be important when indexing
certain characters. For example, ł (U+0142, "LATIN SMALL LETTER L WITH STROKE") does not
have a decomposition mapping in Unicode. Consequently, that character is not considered to have a diacritic.
4 Domain-specific technical jargon often isn't included in the default stemming library. For example, the
word "servlets" should stem to "servlet" but doesn't by default.
68