reindexer

Full-text search with Reindexer

Reindexer has builtin full text search engine. This document describes usage of full text search.

Define full text index fields

Full text search is performed in fields marked with text tag:

type Item struct {
    ID          int64  `reindex:"id,,pk"`
    Description string `reindex:"description,text"`
}

Full text search is also available for multiple fields of composite index marked with text tag:

type Item struct {
    ID          int64  `reindex:"id,,pk"`
    Name        string `reindex:"name,-"`
    Description string `reindex:"description,-"`
    _ struct{}         `reindex:"name+description=text_search,text,composite"`
}

In this example the full-text index will include fields name and description; text_search is a short alias of the composite index name for use in Queries.

The full-text index is case-insensitive. With the default fast splitter, text is tokenized into words: a maximal sequence of letters from a supported Unicode subset, digits 0–9, and symbols from ExtraWordSymbols (default: -/+_`'`). All other characters are delimiters. Word boundaries and extra symbols are configurable; see Text splitters.

Query to full text index

Queries to the full-text index are constructed using the standard query interface.

    query := db.Query ("items").
        Match ("name+description","text query")

Or equivalent query using name alias:

    query := db.Query ("items").
        Match ("text_search","text query")

Queries to the full-text index can be combined with conditions on other fields, e.g.:

    query := db.Query ("items").
        Match ("description","text query").
        WhereInt("year",reindexer.GT,2010)

Each query result contains a match rank. The rank is an integer from 0 to 255, where 0 is the lowest relevancy and 255 is the highest. The query Iterator has a Rank() method, which returns the rank of the current result.

Text query format

Terms and subterms

The format of query is:

query := [@[+]field[^boost][,field2[^boost]]] [=][*]term1[*][~][^boost] [+|-][*]term2[*][~][^boost] ...

Patterns

Field selection

A query may start with a field selector:

@field[,field2...] term

By default, when the same query term matches several fields, the term rank is the maximum rank among those fields.

If SumRanksByFieldsRatio = K > 0 and some fields are marked with +, ranks from the marked fields are added to the winner with decreasing weights:

R = Rmax + K*R1 + K*K*R2 + ...

where R1, R2, … are ranks from + fields sorted from highest to lowest. If the winner field is also marked with +, it is not added twice.

Example: for @f1,+f2,f3,+f4 term, if ranks are R1 < R2 < R3 < R4, then R = R4 + K*R2 (f4 is the winner and f2 is the remaining + field). If R2 < R3 < R4 < R1, then R = R1 + K*R4 + K*K*R2 (f1 is the winner, while f4 and f2 are marked with +).

Binary operators

Terms without an explicit operator are optional and are combined with OR: a document may match any of them.

For example, fox +fast -slow finds documents that must contain fast, must not contain slow, and may also contain fox.

Escape character

Use \ to search for a DSL special character as a literal part of a term. This is useful for characters such as +, -, @, *, ^, ~, =, quotes, or \ itself.

For example, \*crisis searches for the literal word *crisis, not for all words ending with crisis.

The escaped character must be listed in ExtraWordSymbols; otherwise it is still treated as a word separator and will not become part of the indexed term.

A DSL operand may be a phrase enclosed in double or single quotes:

"word1 word2 ..."[~N]

or

'word1 word2 ...'[~N]

The words inside the phrase must appear in the same order. For example, "word1 word2" will not match a document containing only word2 word1.

~N sets the maximum distance, in word positions, between adjacent phrase terms. This argument is optional; the default value is N = 1.

To search for the same phrase words in any order, specify all required permutations explicitly, for example: "word1 word2"~5 "word2 word1"~5.

Synonyms of multiple words are not supported in the phrase.

Examples of text queries

Using select functions

It is possible to use select functions to process result data. For now, you can use snippet, snippet_n and highlight, debug_rank. For composite indexes the result of the function will be written in to corresponding subfields. You can not put [,)\0] symbols in functions params. If the value contains special characters, it must be enclosed in single quotes.

Notice: although text indexes may be created over numeric fields, select functions can not be applied to any non-string field.

For all the functions there are two types of supported syntax with the same behavior: field.func_name(...) and field = func_name(...).

Highlight

This function highlights the text area that was found. It has two arguments -

Example: word: “some text”

b.Query("items").Match("text", query).Limit(limit).Offset(offset).Functions("text.highlight(<b>,</b>)")

result: “some text”

Snippet

Snippet highlights text area and erase other text. It has six arguments - last two is default

Example: word: “some text”

b.Query("items").Match("text", query).Limit(limit).Offset(offset).Functions("text.snippet(<b>,</b>,2,0)")

result: “e text”

Snippet_n

More flexible version of snippet. It has 4 position arguments and 5 named arguments. The named arguments are optional and can be passed in any order.

String values must be enclosed into single quotes.

Parameters’ names may be specified without quotes or in double quotes.

Numbers may be passed without quotes or in single quotes.

Examples: word: “some text string”

b.Query("items").Match("text", query).Limit(limit).Offset(offset).Functions("text.snippet_n('<b>','</b>',2,2,pre_delim='{',post_delim='}',with_area=1)")

result: “{[3,11]e text s}”

b.Query("items").Match("text", query).Limit(limit).Offset(offset).Functions("text.snippet_n('<b>','</b>',5,5,pre_delim='{',post_delim='}',left_bound='o',right_bound='i')")

result: “{me text str}”

Debug_rank

This function outputs additional information about ranking of the found word in the text in the key-value format. Returned format and content may vary depending on reindexer’s version. Works with text-index only.

Example:

b.Query("items").Match("text", "masha").Functions("text.debug_rank()")

result: {term_rank:97, term:маша, pattern:маша, bm25_norm:0.98, term_len_boost:1, position_rank:1, norm_dist:0, proc:100, full_match_boost:0} Маша ела кашу.

What debug_rank fields mean

Field Description
term_rank Score of this subterm occurrence after BM25, position, term length, query boost, and field boost (occurrenceScore in Score of one subterm occurrence).
term Original query term (DSL token) that produced the match.
pattern Actual indexed word or subterm variant that matched (e.g. stemmed form).
bm25_norm Normalized BM25 score for this match: (1 - bm25Weight) + bm25 * bm25Boost * bm25Weight.
term_len_boost Boost from query term length, blended with per-field termLenWeight and termLenBoost.
position_rank Boost from word position in the field (earlier positions score higher).
norm_dist Distance factor between this term and the previous query term in a multi-term or phrase query. Zero for the first term.
proc Base relevancy (subterm.Proc()) of the matched word variant before BM25 and position adjustments. See How term variants are scored.
full_match_boost Extra multiplier when the document contains a full match of the entire query (see How document rank is built).

Merging queries results

It is possible to merge multiple queries results and sort final result by relevancy.

    query := db.Query("items").
        Match("description","text query1")
    q2 := db.Query("another_items").
        Match("description","text query2")
    query.Merge(q2)
    iterator = query.Exec()
    // Check the error
    if err := iterator.Error(); err != nil {
        panic(err)
    }
    defer iterator.Close()
    // Iterate over results
    for iterator.Next() {
        // Get the next document and cast it to a pointer
        switch elem := iterator.Object().(type) {
            case Item:
                fmt.Printf ("%v,rank=%d\n",*elem,iterator.Rank())
            case AnotherItem:
                fmt.Printf ("%v,rank=%d\n",*elem,iterator.Rank())
        }
    }

Natural language processing

Built-in stemmer support is available in full-text search. It enables natural language search of words with same stem. For example, query users will also match user. Stemmer is language specific, so it is necessary to specify language of used stemmer.

All the available stemmers are in this directory.

Typos algorithm

Reindexer handles typos with a language-independent deletion-based algorithm: indexed words and query terms are matched via variants produced by removing characters. Substitutions and extra/missing letters are modeled by comparing these deletion skeletons. MaxTypos limits the total number of deletions counted across the query term and the matched indexed word. For each word, up to MaxTyposInWord = (MaxTypos / 2) + (MaxTypos % 2) characters may be removed when building variants.

Typos apply only to query terms with the ~ suffix. Words shorter than 3 characters are not indexed for typos; words longer than MaxTypoLen (default: 15) are excluded.

Typos handling details

Parameters to tune the algorithm:

Typo matches use base relevancy from Typo and TypoPenalty in Base ranking config; see How term variants are scored.

More examples

MaxTypos = 1 - one symbol may be deleted. black and blaack match if the excess a is deleted in the second word. black and block do not match.

MaxTypos = 2 - up to one symbol may be deleted in each word (2 symbols in total). black and blaack match if the excess a is deleted in the second word. black and block match if a is deleted in the first word and o in the second word. black and blok do not match.

MaxTypos = 3 - up to 2 symbols may be deleted in one word and 1 in the other (3 symbols in total). black and blok match if ac is deleted in the first word and o in the second word.

Configuration

Several parameters of full text search engine can be configured from application side. To set up configuration use db.AddIndex or db.UpdateIndex methods:

...
    ftconfig := reindexer.DefaultFtFastConfig()
    // Setup configuration
    ftconfig.LogLevel = reindexer.TRACE
    // Setup another parameters
    // ...
    // Create index definition
    indexDef := reindexer.IndexDef {
        Name: "description",
        JSONPaths: []string{"description"},
        IndexType: "text",
        FieldType: "string",
        Config: ftconfig,
    }
    // Add index with configuration
    return db.AddIndex ("items",indexDef)

Base config parameters

  Parameter name Type Description Default value
  Bm25Boost float Boost of BM25 ranking 1
  Bm25Weight float Weight of BM25 rank in final rank. 0: BM25 will not change final rank. 1: BM25 will affect final rank in the 0 - 100% range. 0.1
  DistanceBoost float Boost of search query term distance in the found document. 1
  DistanceWeight float Weight of search query term distance in final rank. 0: distance will not change final rank. 1: distance will affect final rank in the 0 - 100% range. 0.5
  TermLenBoost float Boost of search query term length 1
  TermLenWeight float Weight of search query term length in final rank. 0: term length will not change final rank. 1: term length will affect final rank in the 0 - 100% range 0.3
  PositionBoost float Boost of search query term position 1.0
  PositionWeight float Weight of search query term position in final rank. 0: term position will not change final rank. 1: term position will affect final rank in the 0 - 100% range 0.1
  FullMatchBoost float Boost for documents containing a full match of the search phrase 1.1
  PartialMatchDecrease int Penalty for prefix/suffix partial matches: partial_match_decrease * non_matched_symbols / max(matched_symbols, 3), limited by PrefixMin / SuffixMin from Base ranking config 15
  MinRelevancy float Deprecated. Use MinRank instead. Minimum rank of found documents. 0: all found documents will be returned; 1: only documents with relevancy >= 100% will be returned 0.05
  MinRank int Minimum rank of found documents. 0: all found documents will be returned; 255: only documents with relevancy == 255 will be returned 5
  MaxTypos int Maximum typo budget. 0: typos are disabled, words with typos will not match. N: words with N possible typos will match. Check typos handling section for detailed description. 2
  MaxTyposInWord int Deprecated. Use MaxTypos instead. Cannot be used together with MaxTypos. It is not recommended to set more than 1 possible typo per word: it will seriously increase RAM usage and decrease search speed -
  MaxTypoLen int Maximum word length for building and matching variants with typos. 15
  FtTyposDetailedConfig struct Config for more precise typos algorithm tuning  
  MergeLimit int Maximum documents count which will be processed in merge query results. Increasing this value may refine ranking of queries with high frequency words, but will decrease search speed. For a single-term prefix/suffix query the engine may also stop collecting matches after about 2 × MergeLimit document hits 20000
  Stemmers []string List of stemmers to use. More about stemming. Available values: “en”, “ru”, “nl”, “fin”, “de”, “da”, “fr”, “it”, “hu”, “no”, “pt”, “ro”, “es”, “sv”, “tr” “en”,”ru”
  EnableTermsConcat bool Enable concatenated terms processing. e.g. terms “di caprio” will match word “dicaprio” true
  EnableTermsSplit bool Enable splitting query terms. e.g. term “dicaprio” will match words “di” and “caprio”. Also splits a number at the beginning or end of a term, e.g. “season1” produces the “season 1” variant. Terms with ‘-‘ and ‘+’ prefixes will not be split true
  EnableTranslit bool Enable russian translit variants processing. e.g. term “luntik” will match word “лунтик” true
  EnableKbLayout string Wrong keyboard layout variants: "disable", "enable", or "heuristic" (default). See Wrong keyboard layout “heuristic”
  EnableNumbersSearch bool Enable number variants processing. e.g. term “100” may match words “one hundred” false
  StopWords []struct List of stop words. At index time, stop words are not indexed as standalone terms. At query time, behavior depends on is_morpheme; see Stopwords details.  
  SumRanksByFieldsRatio float Ratio used to add ranks when the same term matches several fields selected with + in the field selector 0.0
  LogLevel int Log level of full text search engine 0
  FieldsCfg []struct Configs for certain fields in composite full-text indexes. Overrides parameters from the main config. Contains parameters: FieldName, Bm25Boost, Bm25Weight, TermLenBoost, TermLenWeight, PositionBoost, PositionWeight. empty
  ExtraWordSymbols string Extra symbols that will be treated as word parts in addition to letters and digits. WordPartDelimiters are automatically added to this set. See Text splitters -/+_`'
  KeepDiacritics []string List of symbol types for which diacritics should be kept. Supported values: acc / accent, ara / arabic, heb / hebrew, cyr / cyrillic empty
  Synonyms []struct A list of synonyms to be used in full text search  
  TermsBoost []struct A list of terms boosts to be used in full text search. The relevance of each term will be increased proportionally to the specified boost factor. For each term, all possible word forms generated using all the stemmers listed in Stemmers are boosted  
  MaxAreasInDoc int Max number of highlighted areas for each field in each document (for snippet() and highlight()). ‘-1’ means unlimited 5
  MaxTotalAreasToCache int Max total number of highlighted areas in ft result, when result still remains cacheable. ‘-1’ means unlimited -1
  Optimization string Optimize the index by memory or by cpu. memory uses compressed document id vectors and less RAM; cpu uses uncompressed vectors and may provide faster selection “memory”
  FtBaseRanking struct Base relevancy of term variants before BM25 and field boosts. See Base ranking config and How term variants are scored  
  Bm25Config struct Document ranking function parameters. See Basic document ranking algorithms  
  SplitterType string Text breakdown algorithm. Available values: ‘mmseg_cn’ and ‘fast’ “fast”
  WordPartDelimiters string Symbols that will be treated as delimiters inside words. Delimited parts with at least MinWordPartSize symbols are indexed and searched separately. See Text splitters -/+_`'
  MinWordPartSize int Minimum word part size for indexing and searching delimited word parts 3
  EnablePreselectBeforeFt bool If true, then non-fulltext filtering conditions will be executed before fulltext index selection false

Wrong keyboard layout

EnableKbLayout controls generation of wrong-keyboard-layout variants (for example query keynbr matching indexed лунтик).

Available values:

Additional rules:

Text splitters

Reindexer supports two algorithms to break texts into words: fast and mmseg_cn.

With the default fast splitter, a word is a maximal run of letters from the supported Unicode subset below, digits 0–9, and symbols from ExtraWordSymbols. Everything else (whitespace, punctuation, unsupported Unicode subsets, and so on) is treated as a delimiter.

Reindexer supports the following Unicode blocks and extra symbols:

Symbols from WordPartDelimiters split a word into parts during indexing and search. Each part with at least MinWordPartSize characters is indexed separately, and the word without delimiters is indexed as well. For example, with default delimiters -/+_`' and MinWordPartSize = 3, the text foo-bar is indexed as foo, bar, and foobar.

This algorithm is simple and provides high performance, but it can not handle texts without delimiters (for example, in Chinese, spaces between words are not required, so fast-splitter will not be able to index it properly).

Alternative mmseg_cn-splitter is based on friso implementation of mmseg algorithm and uses dictionaries for tokenization. Currently, this splitter supports only Chinese and English languages.

Stopwords details

Stop words reduce noise from very frequent words. Behavior differs between indexing and querying:

Each list item can be either a string or an object with word and is_morpheme. If the stop word is set as a string, is_morpheme defaults to false. The following entries are equivalent:

"stop_words": [
    {
        "word": "some_word",
        "is_morpheme": false
    }
]

"stop_words": [
    "some_word"
]

Stop words are normalized on load (lowercased, diacritics removed). Spaces are not allowed. Duplicate entries with different is_morpheme values are rejected.

Example

If the stop words list is:

"stop_words": [
    {
        "word": "under",
        "is_morpheme": true
    }
]

and there are two documents: "...under the roof..." and "...to understand and forgive...", then:

With is_morpheme: false, both under and under* are removed from the query and do not participate in search.

If stop_words is omitted from the config, the default_en and default_ru lists are used. All words in these default lists have is_morpheme: true. An explicitly empty stop_words array disables stop words completely.

Detailed typos config

FtTyposDetailedConfig: config for fine-tuning typo correction. These parameters do not increase the total MaxTypos budget; they only add more restrictions to specific typo shapes. For all fields, -1 means “no additional limit”. MaxTypoDistance and MaxSymbolPermutationDistance matter mainly when MaxTypos >= 2. See typos handling for examples.

  Parameter name Type Description Default value
  MaxTypoDistance int Maximum allowed positional shift when a changed symbol in the query variant and the matched indexed word occupy different positions within the deletion-based typo model. Range: [-1, 100]. -1 means no distance limit. See typos handling. 0
  MaxSymbolPermutationDistance int Maximum allowed positional shift for the same symbol when a typo match is explained by moving one letter to another position (for example wsord → sword). Range: [-1, 100]. -1 means no distance limit. See typos handling. 1
  MaxMissingLetters int Maximum allowed difference in deletion count when the indexed word has more deletions than the query typo variant, within the total MaxTypos budget. Range: [-1, 2]. -1 means no additional limit beyond MaxTypos. 2
  MaxExtraLetters int Maximum allowed difference in deletion count when the query typo variant has more deletions than the indexed word, within the total MaxTypos budget. Range: [-1, 2]. -1 means no additional limit beyond MaxTypos. 2

Base ranking config

FtBaseRanking configures the base relevancy (proc) of generated query term variants before BM25, field boosts, position rank, and distance merging are applied. See How term variants are scored and How document rank is built for how these values are applied.

All values are integers in the range [0, 500]. Parameters fall into three groups:

Values above FullMatch are accepted, but coefficient-based variants are capped at 1.0 and therefore cannot become more relevant than the source variant.

  Parameter name Type Description Default value
  FullMatch int Base proc for an exact full word match; normally the baseline for other proc values 100
  ConcatProc int Absolute base proc for concatenated terms, e.g. query di caprio matching indexed word dicaprio 90
  SplitProc int Relevancy coefficient for terms split by EnableTermsSplit, e.g. query dicaprio matching indexed words di and caprio. Each split part gets (parent_proc / 2) * (SplitProc / FullMatch) 90
  PrefixMin int Minimum proc for prefix partial matches after PartialMatchDecrease is applied 20
  SuffixMin int Minimum proc for suffix partial matches after PartialMatchDecrease is applied 10
  Typo int Base relevancy coefficient for typo variants (Typo / FullMatch, capped at 1.0) before TypoPenalty is applied 85
  TypoPenalty int Penalty applied per typo operation. The penalty is scaled down for longer words; final proc is at least 1 15
  StemmerPenalty int Penalty for variants created by stemming. Final proc is at least 1 15
  Kblayout int Base relevancy coefficient for variants generated by wrong keyboard layout correction 90
  Translit int Base relevancy coefficient for transliterated variants 90
  Synonyms int Base relevancy coefficient for synonym variants 95
  Delimited int Relevancy coefficient for query terms split by WordPartDelimiters, e.g. foo-bar → foo and bar (proc * Delimited / FullMatch) 80

For the full ranking pipeline — how variant proc is computed, how term ranks are merged, and how the final 0–255 document score is produced — see fulltext_ranking.md.

Basic document ranking algorithms

Document-level term frequency scoring uses one of the algorithms selected by Bm25Type:

  Parameter name Type Description Default value
  Bm25k1 float k1 saturation coefficient (bm25 and rx_bm25 only) 2.0
  Bm25b float b length-normalization coefficient (bm25 and rx_bm25 only) 0.75
  Bm25Type string Scoring algorithm: rx_bm25, bm25, or word_count “rx_bm25”

bm25 and rx_bm25

Both algorithms use the same general form:

R = IDF * tf * (k1 + 1) / (tf + k1 * (1 - b + b * L / avgL))

The fraction is the BM25 term-frequency component. It grows as tf increases, but with diminishing returns (it saturates toward k1 + 1). Field-length normalization is applied only through the b * L / avgL part of the denominator — not by dividing tf by L.

Formula members:

IDF (inverse document frequency) measures how rare the matched subterm is across the index. It increases the score for terms that appear in fewer documents and lowers the weight of very common terms:

The two BM25 variants differ only in the IDF formula (termCountInDoc — number of occurrences of the subterm in the document field; tf = termCountInDoc for both):

Bm25Type IDF tf
bm25 log(N / (df + 1)) + 1 termCountInDoc
rx_bm25 max(log((N - df + 1) / df) / log(1 + N), 0.2) termCountInDoc

rx_bm25 is the default. Its IDF is normalized by log(1 + N) and floored at 0.2, so very common terms do not collapse toward an extremely low weight.

When to use which:

Length effects are controlled by Bm25b for both types the same way. The practical difference between bm25 and rx_bm25 is usually noticeable mainly through IDF, not through term-frequency scaling.

word_count

The simplest scoring mode: no IDF, no saturation, no length normalization.

R = termCountInDoc

termCountInDoc — number of occurrences of the matched subterm in the document field.

See How document rank is built for how BM25 scores are combined with proc, field boosts, distance, and other factors into the final document rank.

Limitations and known issues

Performance and memory usage

Internally reindexer uses enhanced suffix array of unique words, and compressed reverse index of documents. Typically size of index is about 30%-80% of source text. But can vary in corner cases.

Upsert only stores text; the full-text index is built lazily on the first query to the text field. Indexing uses several threads and is usually fast on multicore CPUs.

After the first build, newly upserted documents are indexed incrementally on the next full-text query. There is no longer a “rebuild steps” setting that trades commit speed for select speed.

On very large texts, lazy indexing can slow down that first query. To avoid this, warm up the index with a dummy query after the last Upsert.

LIKE

LIKE is not a full-text index query. It is a simpler pattern-matching operator for basic text searches: it matches strings against a pattern where _ means any single character and % means any sequence of characters.

    In Go:
    query := db.Query("items").
        Where("field", reindexer.LIKE, "pattern")

    In SQL:
    SELECT * FROM items WHERE fields LIKE 'pattern'

    'me_t' corresponds to 'meet', 'meat', 'melt' and so on
    '%tion' corresponds to 'tion', 'condition', 'creation' and so on
This site is open source. Improve this page.