Apache Solr Reference Guide Covering Apache Solr 6.0

You configure the tokenizer for a text field type in schema.xml with a element, as a child of :
The class attribute names a factory class that will instantiate a tokenizer object when needed. Tokenizer factory
classes implement the org.apache.solr.analysis.TokenizerFactory. A TokenizerFactory's create()
method accepts a Reader and returns a TokenStream. When Solr creates the tokenizer it passes a Reader
object that provides the content of the text field.
Tokenizers discussed in this section:
Standard Tokenizer
Classic Tokenizer
Keyword Tokenizer
Letter Tokenizer
Lower Case Tokenizer
N-Gram Tokenizer
Edge N-Gram Tokenizer
ICU Tokenizer
Path Hierarchy Tokenizer
Regular Expression Pattern Tokenizer
UAX29 URL Email Tokenizer
White Space Tokenizer
Related Topics
Arguments may be passed to tokenizer factories by setting attributes on the element.
The following sections describe the tokenizer factory classes included in this release of Solr.
For more information about Solr's tokenizers, see http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.
Standard Tokenizer
This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter
characters are discarded, with the following exceptions:
Periods (dots) that are not followed by whitespace are kept as part of the token, including Internet domain
names.
The "@" character is among the set of token-splitting punctuation, so email addresses are not preserved
as single tokens.
Note that words are split at hyphens.
The Standard Tokenizer supports Unicode standard annex UAX#29 word boundaries with the following token
types: , , , , and .
Factory class: solr.StandardTokenizerFactory
Apache Solr Reference Guide 6.0
109