Apache Solr Reference Guide Covering Apache Solr 6.0

In: "Visit http://accarol.com/contact.htm?from=external&a=10 or e-mail bob.cratchet@accarol.com"
Out: "Visit", "http://accarol.com/contact.htm?from=external&a=10", "or", "e", "mail", "bob.cratchet@accarol.com"
White Space Tokenizer
Simple tokenizer that splits the text stream on whitespace and returns sequences of non-whitespace characters
as tokens. Note that any punctuation will be included in the tokens.
Factory class: solr.WhitespaceTokenizerFactory
Arguments: rule : Specifies how to define whitespace for the purpose of tokenization. Valid values:
Example:
java: (Default) Uses Character.isWhitespace(int)
unicode: Uses Unicode's WHITESPACE property
In: "To be, or what?"
Out: "To", "be,", "or", "what?"
Related Topics
TokenizerFactories
Filter Descriptions
You configure each filter with a element in schema.xml as a child of , following the element. Filter definitions should follow a tokenizer or another filter definition because they take a T
okenStream as input. For example.
...
The class attribute names a factory class that will instantiate a filter object as needed. Filter factory classes must
implement the org.apache.solr.analysis.TokenFilterFactory interface. Like tokenizers, filters are
also instances of TokenStream and thus are producers of tokens. Unlike tokenizers, filters also consume tokens
from a TokenStream. This allows you to mix and match filters, in any order you prefer, downstream of a
tokenizer.
Arguments may be passed to tokenizer factories to modify their behavior by setting attributes on the e
Apache Solr Reference Guide 6.0
116