Apache Solr Reference Guide Covering Apache Solr 6.0

Understanding Analyzers, Tokenizers, and Filters
The following sections describe how Solr breaks down and works with textual data. There are three main
concepts to understand: analyzers, tokenizers, and filters.
Field analyzers are used both during ingestion, when a document is indexed, and at query time. An analyzer
examines the text of fields and generates a token stream. Analyzers may be a single class or they may be
composed of a series of tokenizer and filter classes.
Tokenizers break field data into lexical units, or tokens.
Filters examine a stream of tokens and keep them, transform or discard them, or create new ones. Tokenizers
and filters may be combined to form pipelines, or chains, where the output of one is input to the next. Such a
sequence of tokenizers and filters is called an analyzer and the resulting output of an analyzer is used to match
query results or build indices.
Using Analyzers, Tokenizers, and Filters
Although the analysis process is used for both indexing and querying, the same analysis process need not be
used for both operations. For indexing, you often want to simplify, or normalize, words. For example, setting all
letters to lowercase, eliminating punctuation and accents, mapping words to their stems, and so on. Doing so
can increase recall because, for example, "ram", "Ram" and "RAM" would all match a query for "ram". To
increase query-time precision, a filter could be employed to narrow the matches by, for example, ignoring all-cap
acronyms if you're interested in male sheep, but not Random Access Memory.
The tokens output by the analysis process define the values, or terms, of that field and are used either to build an
index of those terms when a new document is added, or to identify which documents contain the terms you are
querying for.
For More Information
These sections will show you how to configure field analyzers and also serves as a reference for the details of
configuring each of the available tokenizer and filter classes. It also serves as a guide so that you can configure
your own analysis classes if you have special needs that cannot be met with the included filters or tokenizers.
For Analyzers, see:
Analyzers: Detailed conceptual information about Solr analyzers.
Running Your Analyzer: Detailed information about testing and running your Solr analyzer.
For Tokenizers, see:
About Tokenizers: Detailed conceptual information about Solr tokenizers.
Tokenizers: Information about configuring tokenizers, and about the tokenizer factory classes included in
this distribution of Solr.
For Filters, see:
About Filters: Detailed conceptual information about Solr filters.
Filter Descriptions: Information about configuring filters, and about the filter factory classes included in this
distribution of Solr.
CharFilterFactories: Information about filters for pre-processing input characters.
To find out how to use Tokenizers and Filters with various languages, see:
Language Analysis: Information about tokenizers and filters for character set conversion or for use with
specific languages.
Apache Solr Reference Guide 6.0
104