An analyzer — whether built-in or custom — is just a package which contains three lower-level building blocks: character filters, tokenizers, and token filters.
The built-in analyzers pre-package these building
blocks into analyzers suitable for different languages and types of text.
Elasticsearch also exposes the individual building blocks so that they can be
combined to define new
A character filter receives the original text as a stream of characters and
can transform the stream by adding, removing, or changing characters. For
instance, a character filter could be used to convert Hindu-Arabic numerals
(٠١٢٣٤٥٦٧٨٩) into their Arabic-Latin equivalents (0123456789), or to strip HTML
<b> from the stream.
An analyzer may have zero or more character filters, which are applied in order.
A tokenizer receives a stream of characters, breaks it up into individual
tokens (usually individual words), and outputs a stream of tokens. For
whitespace tokenizer breaks
text into tokens whenever it sees any whitespace. It would convert the text
"Quick brown fox!" into the terms
[Quick, brown, fox!].
The tokenizer is also responsible for recording the order or position of each term and the start and end character offsets of the original word which the term represents.
An analyzer must have exactly one tokenizer.
A token filter receives the token stream and may add, remove, or change
tokens. For example, a
filter converts all tokens to lowercase, a
stop token filter removes common words
(stop words) like
the from the token stream, and a
synonym token filter introduces synonyms
into the token stream.
Token filters are not allowed to change the position or character offsets of each token.
An analyzer may have zero or more token filters, which are applied in order.